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INTRODUCTION 


This  report  is  a  compilation  of  papers  presented  at  the  first  annual  Pikes  Peak 
Educational  Innovations  and  Research  Symposium  (PPEIRS;  pronounced  “peers”),  held  at  the 
US  Air  Force  Academy,  20-22  January  1999.  The  Symposium  was  organized  because  of  the 
following  perceived  problem:  Many  USAFA  faculty  and  staff  were  conducting  "education 
research"  but  knew  little  of  others'  activities  across  academic  departments,  across  academic 
divisions  and  across  organizations  at  USAFA.  The  objective  of  the  Symposium  and  the 
preparation  of  this  report  was  to  allow  USAFA  education  researchers  and  innovators  to 
describe  their  work  to  their  colleagues. 

The  "education  research"  we  discovered  at  USAFA  varied  widely,  from  pedagogical 
research  through  technology  applications  in  teaching  to  studies  aimed  at  curriculum  redesign. 
Similarly,  methods  ranged  Avidely  fi-om  literature  reviews  through  anecdotal  data  acquisition 
to  survey  data  acquisition  and  performance  data  acquisition. 

Early  in  the  Symposium  planning  process,  the  Human-Environmental  Research  Center 
compiled  a  list  of  faculty  and  staff  members  who  were  or  had  been  involved  in  education 
research.  This  list  is  shown  in  the  Appendix  for  reference  and  archival  purposes. 

Session  chairs  for  the  Symposium  were  provided  by  the  USAFA  Faculty  Forum. 

They  included  Lt  Col  Carl  Boedigheimer  (Dept  of  Mathematical  Sciences),  Dr.  Ann  Reagan 
(Dept  of  Philosophy  and  Fine  Arts),  Lt  Col  Russ  Defusco  (Dept  of  Biology),  Lt  Col  Paula 
Britton  (Office  of  the  Registrar),  Dr.  Dan  Pack  (Dept  of  Electrical  Engineering),  Lt  Col  Tony 
Aretz  (Dept  of  Behavioral  Sciences  and  leadership),  Lt  Col  Jim  Pocock  (Dept  of  Civil 
Engineering),  Lt  Col  Pete  Heinz  (Dept  of  Foreign  Languages),  and  Dr.  Shane  Bxuns  (Dept  of 
Physics).  Our  thanks  to  these  individuals  for  a  job  well  done. 

The  papers  in  this  report  are  categorized  in  four  different  tracks:  Pedagogy, 
Information  Technology,  Assessment,  and  Miscellaneous.  Within  tracks,  they  are  listed  in  the 
order,  workshops,  panels,  and  technical  papers,  and  then  by  first  author.  Except  as  noted,  all 
authors  are  faculty,  staff  or  cadets  at  the  US  Air  Force  Academy. 
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PEDAGOGY 


Keynote  Address 

A  Perspective  on  Liberal  Arts  Education 

Timothy  Fuller,  Ph.D. 

Dean,  Colorado  College 
Colorado  Springs,  Colorado 

Abstract:  Much  has  been  written  in  the  last  decade  about  the  rapid  changes  in  society  and  the  need 
for  adaptation  by  our  social  institutions,  including  those  responsible  for  undergraduate  education. 
American  higher  education  is  clearly  rooted  in  the  liberal  arts  model  and  several  of  the  institutions  in 
Colorado  Springs,  including  the  Air  Force  Academy,  provide  modem  variants  of  this  model.  In  this 
presentation,  the  Dean  of  Colorado  College,  a  Liberal  Arts  I  institution,  provided  his  views  on  the 
value  and  future  of  liberal  arts  education  in  the  United  States. 

Colonel  Randy  Stiles,  in  introducing  Dr.  Fuller,  noted  that  the  US  Air  Force  Academy  has  a  core 
curriculum  derived  from  the  liberal  arts  education  model.  He  reminded  us  that  the  founding  fathers  of 
the  Academy  said  things  such  as,  "Be  certain  that  the  graduates  of  the  Academy  have  an  academic 
program  that  brings  in  the  sciences,  brings  in  the  humanities,  and  blends  them  with  their  military 
professional  education.  Make  sure  they  are  prepared  to  be  citizens  of  the  world." 

Dr.  Fuller  emphasized  that  irmovation  in  education  is  important  because  the 
circumstances  under  which  we  conduct  teaching  and  learning  inevitably  change.  On  the  other 
hand,  "There  are  certain  fundamentally  distinguishing  features  of  liberal  learning  which 
persist  through  all  the  contingencies  of  fortunes  and  misfortunes  of  history."  Thus, 
proponents  of  the  liberal  arts  ideail  must  constantly  wrestle  with  what  to  change  and  what  to 
preserve. 

In  his  talk,  he  described  the  distinguishing  features  of  liberal  learning,  identified  five 
challenges  we  face  today,  and  suggested  some  appropriate  responses. 

Because  we  are  creatures  of  reason,  liberal  learning  is  predicated  above  all  on  the 
freedom  of  choice  where  we  can  "imagine  possibilities  for  ourselves  that  are  not  simply 
produced  by  the  natural  environment."  Like  Socrates,  we  have  a  "sense  of  duty  to  choose 
what  is  worthy  to  be  chosen"  and  ultimately  "to  die  in  defense  of  the  examined  life."  Such 
decisions  are  clearly  passionate  as  well  as  rational. 

"A  part  of  what  it  means  to  become  human  carmot  be  satisfied  in  any  other  way  than 
by  reflective  thought,  by  a  life  of  study,  and  by  a  life  of  the  mind.”  An  unbroken  tradition 
exists  of  establishing  various  institutions  to  provide  places  within  society  where  people  can 
“devote  themselves  consciously  and  imdistractedly  to  the  task  of  understanding  and  pursuing 
these  things."  Plato's  Academy,  the  monastic  schools  of  the  early  Middle  Ages,  the  Medieval 
European  universities,  and  the  humanistic  universities  of  the  Renaissance  created  the  study  of 
the  arts  and  sciences  that  we  find  now  throughout  modem  institutions  of  higher  education. 
These  include  schools  of  engineering,  military  academies,  modem  research  universities,  and 
liberal  arts  colleges.  We  are  all  part  of  that  extended  tradition,  one  that  grants  us 
"accumulated  wisdom  and  insight,"  but  that  charges  us  with  the  responsibility  of  tmsteeship; 
"We  must  be  quite  clear  about  what  is  important  to  preserve  and  carry  forward  so  that 
subsequent  generations  will  have  the  same  benefit  of  inheritance  that  we  have.  It  is  always  a 
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matter  of  innovation  and  preservation."  We  must  add  our  voices  to  the  conversation,  but 
voices  that  do  not  expound  "in  a  vacuum  as  if  the  past  did  not  exist." 

Two  general  points  should  enter  into  our  thinking;  (1)  Learning,' which  is  intrinsically 
good,  can  be  an  end  in  itself  unjustified  extrinsically  by  merely  practical  or  utilitarian 
concerns;  and  conversely,  (2)  To  organize  and  make  sense  of  the  world,  learning  always  must 
have  had  "practical,  professional,  and  vocational"  ramifications. 

Liberal  learning,  in  both  these  manifestations,  provides  key  resources.  Because 
"successes  and  disappointments"  come  throughout  a  career,  because  it  is  impossible  for 
"human  beings  to  live  in  only  a  joyful  state,"  we  must  draw  on  self-understanding  and  the 
ability  to  understand  others.  Such  insights  come  not  from  technical  training  and 
specialization,  but  from  "a  deeper  understanding  of  the  human  condition." 

The  challenges  that  face  us  in  higher  education  today  threaten  these  resources,  but 
returning  alumni  with  profound  insights  into  life  and  learning  and  the  continuation  of 
institutions  of  higher  learning  from  ancient  to  modem  times  sustains  faith  in  both  the  value 
and  the  power  of  liberal  learning. 

Our  challenges  can  be  categorized  as  follows:  abstraction,  futurity  or  futurism, 
individualization,  liberation,  and  secularization. 

Abstraction  refers  to  realities  such  as  world  markets,  bureaucracies,  ecological 
systems,  and  mass  media  which,  because  of  their  quantitative  emphasis,  weaken  cohesive 
communities  of  support  once  taken  for  granted.  With  "atomization,"  anxious  people  feel 
abstractly  united  to  one  another  only  through  large,  impersonal  organizations. 

Futurity  or  futurism  refers  to  the  general  processes  of  modernization,  which  focuses 
Otar  attention  away  from  the  past  to  the  future  with  an  emphasis  on  innovation  and  change. 
Distancing  ourselves  from  our  inheritances  as  we  embrace  unknown  but  imagined  future 
possibilities  can  result  in  "disorientation  or  a  loss  of  one's  way"  when  we  neglect  the 
"resources  for  reflection  and  decision  that  might  be  within  our  scope."  Boredom  or 
dissatisfaction  with  self  are  not  valid  excuses  for  turning  only  to  future  alternatives. 

Individualization  can  result  in  an  oscillation  between  demanding  and  protecting  our 
autonomy  and  a  distmstful  longing  for  identification  with  some  kind  of  community. 
"Liberation  from  the  oppressive  or  the  allegedly  oppressive  entities  of  tightly  bound 
communities"  sometimes  results  in  a  lack  of  orientation  or  a  connection  with  the  past. 

Liberation  from  fate,  or  from  fixed  bovmdaries  of  thought  and  action,  is  compounded 
by  the  responsibility  to  choose  "things  that  are  really  worthy  to  be  chosen."  This  complexity 
is  sometimes  ignored  by  undergraduates  who  simply  luxuriate  in  the  ability  to  choose  -without 
recognizing  the  limitations  to  innovational  options. 

Finally,  secularization,  an  antipathy  to  the  dimension  of  transcendence,  can 
circumvent  traditional  liberal  education,  which  has  traditionally  emphasized  a  "sense  of  being 
called  beyond  oneself" 

These  challenges — conditions  which  are  neither  easily  ignored  nor  corrected — 
reinforce  the  need  for  liberal  learning.  Intelligible,  reflective,  and  critical  responses 
emphasize  responsible  patterns  of  thinking  and  an  awareness  of  what  other  human  beings 
have  confronted  in  the  past. 

The  answers  to  the  issues  raised  are  not  easy.  But,  "these  are  matters  to  which  there 
are  potential  intelligent  responses.  The  task  of  liberal  learning  is  to  cultivate  the  capacity  for 
thoughtful  and  knowledgeable  response." 
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Creating  Learning  Organizations  in  Higher  Education:  What?  Why?  And  How?  (Workshop) 

David  B.  Porter,  Ph.D.,  Colonel,  USAF 
Chair,  Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  This  workshop  was  about  understanding  educational  processes  and  organizational  renewal. 
It  helped  attendees  appreciate  how  they  can  contribute  to  organizational  transformation  at  various 
levels  by  creating  authentic  learning  communities.  The  five  disciplines  of  learning  communities 
(systems  thinking,  team  learning,  vision  sharing,  mental  modeling,  and  personal  mastery)  provided  a 
foundation  and  fi-amework  for  presenting  numerous  demonstrations,  examples,  and  mini-exercises. 

The  mission  of  the  USAF  Academy  is  "to  develop  and  inspire  air  and  space  leaders 
with  vision  for  tomorrow."  However,  one  of  the  most  essential  means  to  this  military  end 
involves  the  process  of  integrating  the  core  values  of  the  military  Avith  those  of  higher 
education  and  conveying  them  to  both  faculty  and  students.  External  pressures  caused  by 
strategic  global  changes  world  have  combined  with  internal  factors  to  evoke  significant 
changes  to  educational  structures  and  processes  at  the  Air  Force  Academy  over  the  last 
decade.  Is  it  possible  that  developments  at  this  military  academy  might  be  relevant  to  the 
challenges  faced  by  other  higher  education  institutions? 

In  his  recent  work.  The  Evolvins  Self,  A  Psvcholosv  for  the  Third  Millennium,  Mihaly 
Csikszentmihalyi  makes  a  key  point  about  social  organizations  by  contrasting  an  army  unit 
with  a  typical  university  faculty: 

Social  and  cultural  systems...  can  be  more  or  less  complex  depending  on  how 
differentiated  and  integrated  they  are.  An  army  unit,  for  instance,  is  not  very 
differentiated:  at  each  level  of  the  hierarchy,  individuals  are  more  or  less 
interchangeable.  If  you  are  a  private,  your  identity  may  be  interesting  to  you 
and  your  buddies,  but  as  far  as  the  army  is  concerned  you  are  just  a  nvunber. 

On  the  other  hand,  a  well-run  army  tends  to  be  highly  integrated:  each  fighting 
unit  is  surrounded  by  smoothly  functioning  supply  lines,  medical  services,  and 
communication  networks.  Whatever  happens  to  one  unit  has  immediate 
consequences  for  all  the  others,  and  produces  an  adaptive  response  from  them. 

A  typical  university  is,  in  many  ways,  at  the  opposite  extreme:  each  member  of 
the  faculty  operates  in  splendid  isolation  from  his  or  her  peers;  the  emphasis  is 
on  original  accomplishment  and  individuality,  with  little  sharing  of 
information  or  mutual  assistance.  It  is  in  fact  quite  rare  to  find  social 
institutions  that  maximize  complexity  by  being  both  differentiated  and 
integrated  simultaneously,  and  when  they  are,  they  usually  are  so  only  for  a 
short  time,  after  which  they  again  become  either  excessively  rigid,  or  too 
xmstructured.  (p.  253) 

Peter  Senge  offers  learning  organizations  as  a  unique  blend  of  integration  and 
differentiation.  He  goes  on  to  defined  such  groups  as  "organizations  in  which  people  at  all 
levels  are  collectively  and  continually  enhancing  their  capacity  to  create  what  they  really 
want".  Despite  the  variety  of  diverse  examples  of  learning  institutions,  organizations  and 
communities,  they  all  share  an  appreciation  and  incorporation  of  five  necessary  disciplines. 

(A  discipline  is  simply  a  practice  or  exercise  that  builds  capacity  over  time.)  These  five 
disciplines  are  systems  thinking,  mental  models,  sharing  vision,  team  learning  and  personal 
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mastery.  Each  of  them  is  necessary  and  together  they  are  sufficient  to  create  and  sustain 
learning  communities. 

Systems  Thinking,  Peter  Senge's  famous  Fifth  Discipline  stresses  the  importance  of 
understanding  contexts  and  relationships.  This  first  disciple  opens  the  door  to  understanding 
and  appreciating  the  other  disciplines;  it  represents  the  broadest  level  of  conceptualization. 
Systemic  models  of  both  educational  and  organizational  transformation  will  be  used  to 
illustrate  the  advantages  and  insights  provided  by  a  systems  perspective.  Differences  between 
education  and  training  and  their  implications  for  assessment  and  continuous  improvement 
wall  be  briefly  presented. 

Team  Learning  involves  learning  about  the  growth  and  development  of  groups  and 
communities  as  interdependent  wholes.  Building  upon  what  participants  learned  about 
systems,  they  will  consider  how  groups  change.  Characteristics  of  four  distinctive  stages  of 
group  development  (forming,  storming,  norming,  and  performing)  will  show  many  of  the 
potential  difficulties  inherent  in  the  creation  of  effective  teams.  Discussion  of  the 
consequences  of  creating  competitive  vs.  cooperative  systems  also  will  be  presented  briefly. 

Mastery  of  the  discipline  of  Sharing  Vision  requires  the  active  and  enthusiastic 
participation  of  all  individual  stakeholders  in  the  organization.  Traditional  top-down 
approaches  are  unlikely  to  yield  the  quality  of  product  or  earn  the  ubiquitous  individual 
commitment  necessary  to  sustain  systemic  improvement.  Conversely,  a  leader's  failure  to 
initiate  the  appropriate  conversations  about  what  is  most  important  can  lead  to  apathy  and 
confusion.  Some  unique  characteristics  of  the  human  visual  processing  system  provide 
interesting  metaphors  for  the  development  of  vision  sharing  in  educational  organizations. 

Mental  Models,  the  penultimate  discipline,  reveals  some  of  the  underlying  cognitive 
structures  that  profoundly  influence  our  perception,  thought  and  behavior.  By  articulating  and 
sharing  graphic  representations  of  these  structures,  attendees  will  better  rmderstand  their  own 
and  others'  views  of  educational  processes.  An  understanding  of  these  mental  models  and  the 
role  they  play  in  all  human  thought  and  memory,  reveals  the  importance  of  frequently 
examining  them  and  sharing  them  with  others. 

The  discipline  of  Personal  Mastery  involves  the  appreciation  of  general  principles  of 
human  growth  and  development  as  well  as  an  imderstanding  of  one's  own  particular  and 
distinctive  strengths,  weaknesses  and  preferences.  Previous  research  has  shown  that  a  variety 
of  distinctive  learning  styles  and  alternative  approaches  to  a  wide  variety  of  educational 
activities  can  enhance  organizational  learning.  This  portion  of  the  workshop  will  discuss 
implications  of  individual  differences  for  comfort  in  a  variety  of  educational  activities 
including  teaching,  learning,  and  outcomes  assessment. 

Commitment  to  pursuing  all  five  of  these  disciplines  will  inevitably  increase  an 
organization's  vmderstanding  of  its  ultimate  purposes  and  awareness  of  its  most  important 
processes.  Pursuit  of  these  disciplines  also  serves  as  a  catalyst  for  continuous  learning  and 
quality  improvement.  Educational  organizations  are  especially  likely  to  benefit  from  adopting 
this  broad  conceptual  framework. 
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The  Collegial  Classroom:  DFBL's  Efforts  to  Learn  About  Learning  and  its  Implications  for 

Classroom  Teaching  (Panel) 

Robert  C.  Berger,  Ph.D.,  Lt  Colonel,  USAF 
Department  of  Behavioral  Sciences  and  Leadership 

Panel  members:  Col  David  B.  Porter,  Maj  Kristen  Vance,  Dr.  Steve  Samuels,  CIC  James  M. 

Ferrell,  CIC  Kari  A.  Hamilton,  and  CIC  Andrew  M.  Quinn 
Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  Over  the  past  decade,  many  faculty  members  in  USAFA's  Department  of  Behavioral 
Sciences  and  Leadership  have  been  actively  seeking  ways  to  enhance  curriculum  and  pedagogy  to 
enhance  student  learning.  Recently,  four  particular  pedagogical  approaches  have  received  increasing 
support  within  the  department:  curricular  modularity,  quiz-first,  accept-resubmit  enrichment  projects, 
and  benefit  points.  Together,  these  four  approaches  contribute  to  the  development  of  a  productive  and 
encouraging  atmosphere  we  refer  to  as  the  collegial  classroom.  This  panel  was  comprised  of  students 
and  teachers  who  had  first  hand  experience  with  these  approaches  and  their  effects. 

Over  the  past  decade,  many  faculty  members  in  the  US  Air  Force  Academy's 
Department  of  Behavioral  Sciences  and  Leadership  have  been  developing  curriculum  and 
pedagogy  that  enhance  student  learning.  Many  different  initiatives  have  been  explored. 
Collaborative  and  cooperative  learning,  interdisciplinary  education,  curricular  integration, 
affective  engagement,  distributed  practice,  and  embedded  assessment  have  all  been  shown  as 
ways  of  enhancing  student  motivation  and  learning.  The  overarching  principle  of  this  work  is 
to  move  from  a  traditional  teaching  focus  to  more  of  a  learning  focus  (Barr  &  Tagg,  1995). 
That  is,  moving  from  a  point  where  the  instructor  is  seen  as  the  source  of  a  one-way  flow  of 
knowledge  to  the  students,  and  moving  toward  a  point  where  the  instructor  is  seen  as  a  guide 
and  facilitator  to  the  students’  seeking  and  attaining  their  own  knowledge.  This  shift  in 
approach  turns  the  classroom  into  a  more  collegial  setting  by  giving  students  more 
responsibility  for  their  education,  where  the  goal  is  a  mutual  learning  environment  for  both 
teacher  and  student.  This  paper  addresses  four  particular  pedagogical  approaches  that  have 
been  put  into  practice  within  the  department:  curricular  modularity,  quiz-first,  accept/improve 
papers,  and  benefit  points. 

Curricular  modularity  is  the  term  used  to  describe  an  approach  to  course  design  based 
on  the  integration  of  topics  into  a  coherent  story.  Over  the  long  term,  stories  are  much  more 
memorable  than  lexicons  or  random  collections  of  isolated  facts  and  procedures.  All  too  often, 
text  availability  is  the  primary  determinant  of  course  structure.  There  are  always  many  texts 
available,  but  what  makes  a  profitable  or  popular  textbook  is  not  necessarily  the  same  thing 
that  makes  for  a  coherent  or  effective  course.  In  fact,  the  market  pressures  Aat  push  texts  to 
endeavor  to  be  (and  thus  include)  all  things  for  all  people  and  purposes,  make  many  of  them 
unsuitable  as  a  framework  for  course  design.  By  focusing  first  on  broad  themes,  then  on 
interlocking  topics  that  support  those  themes,  courses  can  be  constructed  before  texts  are 
selected.  A  text  can  then  be  chosen  (or  created)  that  supports  the  story  the  course  hopes  to 
convey.  The  course  director  can  then  select  supplemental  readings  and  enrichment  projects  to 
increase  course  coherence  and  coverage  as  needed. 

No  matter  how  good  a  text  is,  it  will  not  contribute  to  students'  learning  unless  they 
read  it  and  reflect  on  the  information  it  contains.  The  quiz-first  approach  is  a  useful  method 
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of  using  just  enough  pressure  to  bring  about  general  compliance  with  the  course's  classroom 
preparation  requirements.  Although  there  are  various  alternative  forms  of  this  approach,  in 
general  each  module  or  block  of  material  begins  with  a  quiz.  Quizzes  are  relatively  brief, 
comprised  of  questions  that  tap  material  the  student  should  have  gleaned  from  reading  the 
chapter.  The  quiz  is  completed  in  a  single  period  and  usually  involves  four  distinct  phases. 
First,  students  are  given  the  opportunity  to  ask  any  questions  about  topics  covered  in  the 
reading  that  were  unclear  or  confusing.  Next,  they  take  the  quiz  individually.  As  soon  as  they 
have  completed  their  individual  quizzes,  they  form  into  groups  of  3  or  4  students  and  retake 
the  same  quiz  producing  a  single  answer  sheet.  Finally  the  whole  class  convenes  and 
discusses  the  quiz.  Questions  can  be  dropped  due  to  ambiguity  or  credit  can  be  given  for 
multiple  responses  if  appropriate.  The  grade  students  receive  for  a  quiz  is  a  weighted  average 
of  their  individual  score  and  their  group's  score.  Note  they  can  either  gain  or  lose  points  based 
on  their  group's  performance.  While  the  quiz  takes  an  entire  period  of  class,  all  students  (even 
the  few  who  arrived  relatively  unprepared)  have  been  engaged  in  substantive  discussions  of 
the  most  important  aspects  of  the  module  topic  by  end  of  the  50  minutes.  This  leaves  several 
more  lessons  for  elaboration,  experience,  and  application  of  the  material,  especially  targeting 
that  information  that  was  least  imderstood  by  the  students. 

Another  example  of  collegiality  is  the  accept/improve  paper.  Instead  of  assigning  a 
letter  or  numerical  grade,  short  papers  now  need  to  be  "accepted".  Those  that  demonstrate 
learning  and  are  of  excellent  quality  are  accepted;  those  that  are  not  receive  feedback 
concerning  needed  improvements.  This  feedback  is  often  in  the  form  of  questions  the  teacher 
would  like  answered  based  on  the  topics  examined.  Students  may  resubmit  papers  as  many 
times  as  necessary  to  demonstrate  learning  and  excellence.  The  final  paper  grade  is  based  on 
the  number  of  papers  accepted  over  the  semester.  For  example,  a  course  could  have  10%  of 
the  overall  grade  set  aside  for  papers.  Each  of  the  first  three  accepted  papers  could  count  for 
thirty  points,  while  the  fourth  could  count  for  ten  points.  Those  students  who  wanted  to  do 
the  minimal  work  must  have  two  papers  accepted  (60%  of  the  possible  credit).  Those  who 
wish  to  put  forth  more  effort  would  ensure  three  papers  were  accepted  (90%),  and  those  who 
really  wanted  to  put  in  maximal  effort  would  work  on  all  fotir  (100%).  This  system  allows 
students  to  demonstrate  intellectual  curiosity  by  exploring  what  they  find  intrinsically 
interesting. 

Finally,  benefit  points  can  be  used  to  place  some  the  responsibility  for  assessment 
directly  on  the  student.  Somewhat  like  class  participation  points,  benefit  points  are  assigned 
based  on  how  much  the  student's  presence  benefited  the  class.  Talking  too  much 
(dominating)  is  not  rewarded;  nor  is  talking  without  saying  anything.  Listening  and  actively 
reacting  to  other  students  is  ideal.  Points  can  also  be  earned  in  other  ways  that  benefit  the 
class  (e.g.,  feedback  to  the  teacher  about  the  course,  feedback  to  other  students  by  reading 
their  papers,  exploring  topics  outside  of  class  and  reporting,  etc.).  The  assessment  is  made 
half  by  the  student  and  half  by  the  teacher.  Both  are  blind  assignments  so  that  neither  can 
"adjust"  their  score  to  take  advantage  of  the  other.  The  mutual  assignment  takes  place  at  mid¬ 
term  and  final,  and  is  non-cumulative.  This  allows  students  to  learn  to  improve  from  their 
mid-term  feedback,  and  does  not  allow  those  doing  well  to  coast  in  the  second  half  It  is 
important  to  note  that  the  assessments  are  not  merely  subjective,  but  based  on  structured 
guidelines:  clear  expectations  and  explanations,  possibly  with  handouts  or  behavioral  profiles 
of  what  point  assignments  would  look  like.  Additionally,  any  assessment  over  70%  must  be 
justified  by  the  student  in  a  short  correspondence  to  the  teacher  explaining  why  she  or  he 
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deserves  the  points  (which  can  also  serve  as  a  form  of  course  feedback).  Then,  teachers 
return  the  correspondence  with  their  own  assessment  explanation  of  the  student,  effectively 
re-focusing  them  on  both  how  he  or  she  did  and  what  is  expected  after  ihid-term.  This 
approach  increases  students'  perceived  control  in  class,  their  self-efficacy  for  learning  and 
helps  lead  them  to  believe  they  are  being  treated  fairly. 

All  four  methods  we  have  discussed  are  consistent  with  the  idea  of  moving  from  a 
teaching  to  a  learning  paradigm.  These  methods  give  students  a  significant  amount  of  control 
over  their  grade  and  increase  their  responsibility  for  their  own  learning.  Course  critique  data, 
majors'  exit  surveys,  and  graduation  survey  data  suggest  that  collegial  classroom  techniques 
have  a  positive  effect  on  learning  and  student  enjoyment. 

Reference 

•  Barr,  R.  B.,  &  Tagg,  J.  (1995).  A  new  paradigm  for  undergradrmte  education.  Change. 
November/December,  13-25. 
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Helping  Cadets  Develop  Their  Skills  for  Framing  and  Resolving  Ill-Defined  Problems 

Cindy  Lynch,  Ph.D. 

Consultant 

New  Concord,  Kentucky 

Abstract:  Faculty  in  engineering,  legal  studies,  behavioral  sciences,  and  other  disciplines  at  the 
USAFA  have  been  using  innovative  strategies  to  assess  student  performance  and  to  help  their  students 
develop  skills  for  framing  and  resolving  ill-defined  problems.  Such  problems  do  not  have  absolutely 
correct  answers,  and  experts  often  disagree  about  optimal  solutions.  In  this  session,  faculty  shared 
some  of  what  they  had  learned  about  students’  skills  and  offered  suggestions  for  classroom 
implementation  of  this  cognitive  developmental  approach. 

Over  the  past  four  years,  several  Air  Force  Academy  faculty  members  have  used  a 
developmentally-grounded  problem  solving  process  as  the  foundation  for  helping  students 
learn  how  to  frame  and  resolve  ill-defined  problems.  In  contrast  to  well-defined  problems 
that  have  a  single,  correct  answer,  ill-defined  problems  are  fi-aught  with  significant  and 
enduring  uncertainties  and  ambiguities  that  preclude  a  single,  correct  answer.  Based  on  a 
panel  discussion  of  faculty  experiences  in  using  the  problem  solving  model,  several  benefits 
were  identified: 

•  Process  provides  opportunities  to  address  a  wide  range  of  learning  outcomes 

•  Very  adaptable;  direct  faculty  involvement  in  the  design  of  activities  and  evaluation  of 
performance 

•  Relatively  efficient;  provides  both  learning  opportunities  for  students  and  assessment  data 
for  educators 

•  Theoretically  and  empirically  grounded  in  human  development  so  it  provides  a  way  to 
understand  current  performance  and  developmental  steps  between  current  performance 
and  desired  performance 

•  Dynamic  skill  theory  explicitly  addresses  the  interplay  of  personal  and  environmental 
characteristics;  it  has  direct  implications  for  curriculum  refinements 

Faculty  also  identified  some  common  concerns  about  using  this  developmental  approach. 
Some  of  the  most  important  concerns  are  addressed  below. 

Concern:  I’m  not  a  developmental  psychologist,  and  I’m  unsure  about  how  all  this  works. 
Suggestion:  For  more  information  about  assessing  and  optimmng  students’  skills  for  framing 
and  resolving  ill-defined  problems,  see  A  Developmental  Guide  to  Assessing  and  Optimizing 
Professional  Problem  Solving  (available  electronically  at  <http://www.du.edu/~swolcott>). 
Engage  in  the  problem  solving  process.  Discuss  your  ideas  with  a  colleague  or  Cindy  Lynch. 
Make  some  minor  change  and  see  what  happens. 

Concern:  Won’t  it  be  very  time  consuming  to  deal  with  each  student’s  individual  needs? 
Suggestion:  Again,  start  small.  Be  very  aware  of  your  own  time  constraints  and  those  of  your 
students.  What  might  you  delete  from  the  work  load  to  allow  some  time  for  innovative 
approaches?  As  you  work  ■with  this  model,  the  skill  patterns  ■will  become  more  clear  to  you 
and  you  will  become  more  efficient  in  recognizing  and  addressing  individual  needs. 
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Concern:  Doesn’t  cognitive  development  take  a  long  time?  What  ifl  can’t  demonstrate 
improvements  during  a  semester? 

Suggestion:  Can  you  already  demonstrate  improvements  in  framing  and  resolving  ill-defined 
problems  during  a  semester?  Most  naturalistic  environments  do  not  provide  the  structure  to 
help  students  exhibit  their  best  performance.  Providing  the  problem  solving  “roadmap”  and 
opportunities  for  reflection  and  feedback  can  help  students  exhibit  stronger  performance  and 
further  develop  their  skills,  even  over  the  time-span  of  a  semester. 

Concern:  I  want  my  students  to  learn  to  take  a  stand  and  defend  their  opinion.  Won’t  all  this 
analysis  just  confuse  them  and  give  them  an  excuse  to  avoid  a  decision? 

Suggestion:  Many  students  learn  to  “play  the  game”  of  giving  faculty  what  they  think  we 
want  to  see.  They  learn  to  “stack  up”  evidence  to  support  a  particular  point  of  view. 

Although  this  is  a  precursor  to  the  more  complex  framing  sldlls,  it  is  not  sufficient.  We  may 
be  encouraging  superficial  analysis.  Ask  your  students  to  respond  to  some  framing  reflection 
prompts  and  see  how  they  perform. 

Concern:  How  will  students  respond  to  what  I  ask  them  to  do? 

Suggestion:  While  many  students  respond  very  positively  to  having  this  “roadmap”  and 
become  more  engaged  in  learning,  some  students  complain  that  the  tasks  are  too  difficult  or 
that  their  current  way  of  thinking  suits  them  just  fine  (a  “nobody  knows  so  anything  goes” 
approach).  Emotions  organize  development;  feelings  can  inhibit  or  motivate  action.  Attend 
to  students’  concerns.  Provide  clear  instructions  and  examples.  Conforming  often  precedes 
understanding.  Listening  and  providing  appropriate  feedback  is  important.  Adjust  what  you 
are  doing  as  needed. 

Reflective  judgment  research  suggests  that  students  with  different  skill  patterns  are  likely  to 
hold  certain  beliefs  that  hinder  their  performance  (see  Developmental  Guide  for  lists  of 
beliefs).  These  beliefs  impact  how  they  are  likely  to  respond  to  ill-defined  problems.  Here 
are  two  common  response  patterns  among  undergraduates: 

•  Weak  identifying,  framing,  resolving,  and  re-addressing:  Students  operating  at  this  level 
expect  “experts”  to  provide  them  with  all  answers,  and  thus  are  not  likely  to  exhibit 
initiative  in  developing  their  own  solutions  to  ill-defined  problems. 

•  Adequate  identifying,  but  weak  framing,  resolving,  and  re-addressing:  Students  operating 
at  this  level  tend  to  be  open  to  new  situations,  and  they  no  longer  expect  experts  to  solve 
problems  for  them.  However,  they  tend  to  be  rather  limited  in  their  understanding  of 
problems.  While  they  assert  that  they  are  open  to  others’  viewpoints,  they  often  stack  up 
support  for  their  own  position  without  giving  careful  consideration  to  alternatives. 

If  you  would  like  more  information  about  this  developmental  approach,  please  contact 
Dr.  Cindy  Lynch  via  e-mail  (Leehaven@compuserve.com)  or  contact  these  USAFA  faculty 
members:  Lt  Col  Tony  Aretz,  Lt  Col  Jeff  Jackson  (Dept  of  Behavioral  Sciences  and 
Leadership);  Maj  Monty  Greer,  Maj  Paul  Waters  (Dept  of  Engineering  Mechanics);  Lt  Col 
Donna  Peterson,  Maj  Cameron  Wright  (Dept  of  Electrical  Engineering);  or  Maj  Pam  Perry 
(Dept  of  Law). 
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Igniting  Intellectual  Curiosity  with  "ICE" 

Robert  C.  Berger,  Ph.D.,  Lt  Colonel,  USAF  • 

Steven  K.  Jones,  Ph.D. 

Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  One  of  the  Academy's  educational  outcomes  is  to  develop  officers  who  are  "intellectually 
curious."  This  paper  briefly  outlines  a  current  theory  of  curiosity  and  explores  assessment  efforts  that 
examine  how  well  the  Academy  is  meeting  its  outcome  of  graduating  officers  who  are  intellectually 
curious.  We  describe  a  course  project  designed  to  enhance  intellectual  curiosity  that  we  have  used  for 
three  years  in  one  of  our  Behavioral  Science  courses  and  present  student  critique  data  evaluating  the 
success  of  our  project. 

One  of  the  United  States  Air  Force  Academy's  educational  outcomes  is  to  develop 
officers  who  are  "intellectually  curious."  The  outcome  further  states,  "Beyond  possessing 
knowledge  and  having  abilities  to  put  that  knowledge  to  active  use,  graduates  of  the  Academy 
must  be  inclined  to  do  so.  We  want  to  develop  an  attitude  of  intellectual  curiosity  in  our 
graduates  that  predisposes  them  to  lifelong  learning"  (United  States  Air  Force  Academy 
Educational  Outcomes,  1997).  The  Academy  is  not  alone  in  its  quest  for  developing  curiosity 
in  its  students.  Curiosity  has  been  described  as  being  crucial  to  educational  attainment  (Day, 
1982),  and  teachers  are  routinely  encouraged  to  stimulate  curiosity. 

Over  100  years  ago,  William  James  (1890/1950)  proposed  that  "scientific  curiosity" 
arises  from  an  "an  inconsistency  or  a  gap  in  ...knowledge"  (p.429).  James'  early  ideas  about 
curiosity  foreshadowed  a  current  psychological  theory  of  curiosity — ^the  "information-gap" 
theory  (Loewenstein,  1994).  The  information-gap  theory  takes  the  perspective  that  curiosity 
arises  when  one's  attention  becomes  focused  on  a  gap  in  one's  knowledge.  The  theory  states 
that  curiosity  is  positively  related  to  knowledge  in  a  specific  domain.  As  a  student  gains 
knowledge  in  a  specific  area  it  becomes  more  likely  that  the  student  will  focus  on  what  they 
do  not  know  rather  than  what  they  do  know.  Therefore,  it  is  not  surprising  that  as  knowledge 
in  a  domain  increases,  so  does  curiosity  (Loewenstein,  1994). 

These  ideas  make  the  pursuit  of  curiosity  particularly  appropriate  for  a  major's  course. 
Although  some  students  may  seek  out  situations  diat  may  make  them  curious,  a  more 
successful  strategy  in  our  fast-paced,  demanding  environment  at  the  Academy  may  be  to 
deliberately  expose  students  to  potentially  curiosity-inducing  stimuli.  The  Intellectual 
Curiosity  Exercise  (ICE)  in  Behavioral  Sciences  336-Cognitive  Psychology  is  designed  to 
spark  students'  curiosity  by  exposing  them  to  more  in-depth  information  in  their  chosen  major 
field  of  study  by  requiring  them  to  read  a  book  related  to  cognitive  psychology.  Based  on 
their  reading  of  the  book,  they  make  a  oral  presentation  to  the  other  members  of  their  class, 
and  they  also  complete  a  written  paper. 

The  Academy  has  recently  assessed  how  well  the  academic  program  has  addressed  the 
fostering  of  intellectual  curiosity  (United  States  Air  Force  Academy  Educational  Outcomes 
Assessment  Working  Group  (EOAWG).  Phase  I:  Final  Report,  November  1995).  In  a 
"Faculty  Practices  Survey,"  93%  of  the  faculty  said  that  developing  intellectual  curiosity  is 
either  "important"  or  "very  important"  (EOAWG  Report).  The  report  also  suggested  several 
interventions  that  would  help  foster  intellectual  curiosity.  These  suggestions  included 
incorporating  more  choice  for  students,  more  use  of  cooperative  learning  techniques,  and 
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maintaining  a  more  relaxed  classroom  atmosphere  with  more  opportunity  for  personal 
interaction  with  instructors. 

We  also  found  during  informal  conversations  with  cadets  that  many  admitted  to 
having  never  read  a  non-fiction  book  that  was  not  required  for  class,  and  that  many  cadets  did 
not  even  read  for  pleasure.  We  also  discovered  that  for  many  of  our  students  the  only 
exposure  to  their  chosen  major  was  through  course  texts  and  classroom  lectures.  We  felt 
there  were  many  contemporary  books  in  the  Behavioral  Sciences  area  (some  on  recent  best¬ 
seller  lists,  like  The  Bell  Curve  by  Richard  Hemnstein  and  Charles  Murray,  and  Emotional 
Intelligence  by  Daniel  Goleman)  that  our  students  would  benefit  from.  We  also  thought  that 
exposing  them  to  reading  about  their  major  in  a  less  structured,  more  informal  setting,  might 
lead  to  more  spontaneous  reading  on  their  part.  We  felt  that  this  in-depth  reading  could 
potentially  highlight  gaps  in  our  students  knowledge  and  increase  their  curiosity  about 
psychology. 

We  developed  the  Intellectual  Curiosity  Exercise  (ICE)  which  we  have  used  for  three 
semesters  in  Behavioral  Science  336-Cognitive  Psychology.  An  excerpt  from  the  syllabus 
follows: 

The  ICE  is  one  of  the  most  unique  and  exciting  aspects  of  this  course.  The  goal  is  to 
give  you  the  opportunity  to  read  a  book  of  your  own  choosing  on  a  topic  related  to  file  field  of 
cognitive  psychology.  You  ivill  then  give  a  short  presentation  about  the  book  to  the  class,  as 
well  as  write  a  paper  about  what  you  have  learned.  This  process  will  give  you  the  chance  to 
think  deeply  about  the  book  that  you  read,  and  it  will  also  give  you  the  chance  to  learn  at  least 
a  little  bit  about  the  books  read  by  your  classmates.  There  are  really  three  required  elements 
of  the  Intellectual  Curiosity  Exercise.  They  are:  (1)  Choosing  and  reading  a  book;  (2)  Giving 
a  class  presentation;  and  (3)  Writing  a  paper  about  what  you’ve  learned.  (Behavioral  Sciences 
336  Syllabus,  Spring  1999) 

We  provide  the  students  with  a  suggested  list  of  books,  but  allow  books  not  included 
on  the  list  to  be  approved  by  the  instructors.  The  exercise  is  modeled  by  course  instructors 
who  read  a  book  from  the  project  list  and  make  an  in-class  presentation  early  in  the  semester. 
Students  are  encouraged  to  make  their  presentations  informal  and  interactive.  To  complete 
the  ICE,  students  schedtrle  a  personal  appointment  with  their  instructor  to  discuss  what  they 
learned  from  the  project  and  to  support  their  recommendation  for  a  grade. 

The  ICE  has  a  number  of  positive  aspects  that  are  consistent  vdth  the  information-gap 
theory  of  curiosity  and  with  the  recommendations  of  the  EOAWG  Report.  The  project  builds 
upon  foundational  knowledge  by  requiring  them  to  read  an  in-depth  treatment  of  a  topic  in 
cognitive  psychology.  This  increases  the  likelihood  that  the  student  will  recognize  a  gap  in 
their  knowledge,  and  hopefully  will  encourage  them  to  address  that  gap.  The  exercise  also 
gives  students  practice  in  distilling  and  communicating  complex  arguments  through  both  the 
written  and  ord  presentations.  Finally,  a  number  of  characteristics  of  the  ICE  address  issues 
related  to  those  discussed  in  the  EOAWG  report:  (1)  choice  is  increased  because  students  are 
allowed  to  choose  their  project;  (2)  students  are  actively  involved,  especially  during  their  own 
oral  presentation;  (3)  the  classroom  atmosphere  during  the  classroom  presentation  is  relaxed 
and  open;  and  (4)  the  debriefing  meeting  that  students  have  with  their  instructor  allows  for 
increased  personal  interaction  between  the  student  and  the  insttuctor. 

We  are  currently  using  the  ICE  project  for  the  third  semester  in  cognitive  psychology. 
We  plan  to  develop  more  formal  assessments  of  the  project,  but  thus  far  student  feedback  has 
been  encouraging.  Critique  data  has  shown  that  77%  of  students  agreed  that  the  project 
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should  be  retained  within  the  course  and  that  77%  agreed  that  the  project  enhanced  their 
intellectual  curiosity.  Enhancing  students'  intellectual  curiosity  is  a  specific  goal  of  the 
Academy,  and  tihe  ICE  project,  designed  to  be  consistent  with  current  theorizing  about 
curiosity  and  incorporating  recommendations  from  the  EOWAG  report,  is  a  positive  step 
toward  addressing  that  outcome. 
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"Quiz-First."  An  Anecdotal  and  Empirical  Look  at  a  New  Teaching  Pedagogy 

Joseph  C.  Hickox,  M.S.,  Major,  USAF 
Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  This  paper  addresses  a  new  teaching  methodology  being  advocated  and  used  quite 
extensively  in  the  Department  of  Behavioral  Sciences  and  Leadership  at  the  USAF  Academy. 

This  methodology  uses  a  "quiz-first"  prior  to  teaching  a  block  of  material,  in  essence  forcing 
the  student  to  prepare  for  an  assessment  prior  to  an  instructor  led  lecture.  An  anecdotal  and 
empirical  analysis  of  one  class  taught  two  different  semesters  using  both  a  traditional 
lecture/exam  format  and  a  "quiz-first"  format  is  presented.  Both  pro  and  con  data  will  be 
presented,  but  a  preference  (in  this  class)  for  "quiz-first"  method  is  advocated. 

Introduction 

At  most  institutions  of  higher  learning,  measurement  of  student  learning  in  the 
classroom  and  assignment  of  grades  has  traditionally  been  made  by  the  use  of  tests  or  exams 
given  at  periodic  intervals  throughout  the  semester  or  quarter.  The  exams  typically  covered 
material  that  was  in  a  text  and  had  been  presented  first  in  a  classroom  setting  either  by 
lectures,  demonstrations,  or  student-led  presentations.  This  paradigm  has  been  around  for 
many  years  and  has  been  seldom  challenged  as  the  most  effective  method  of  assessing  student 
learning.  Samuels  (1997)  outlines  a  new  methodology  that  radically  changes  this  model.  He 
proposes  a  "quiz-first"  methodology  that  does  away  with  traditional  lecture-based  exams,  and 
instead  has  as  its  core  assessment  technique,  quizzes  that  are  given  prior  to  covering  the 
material.  Instead  of  exams  that  cover  large  portions  of  the  material,  there  would  be  several 
smaller  quizzes  administered  prior  to  going  over  it  in  class.  Following  the  quizzes  is  more 
traditional  presentation  of  material.  Samuels  gives  further  supporting  evidence  as  to  the 
efficacy  of  this  system.  The  focus  of  this  paper  is  to  report  anecdotal  and  empirical  evidence 
accumulated  by  me  suggesting  that  this  system  is  not  only  a  very  robust  tool  for  assessing 
learning,  but  also  encourages  student  learning  more  than  the  traditional  approach. 

I  adopted  this  approach  while  course  directing  and  teaching  a  class  in  Engineering 
Psychology  and  Human  Performance.  There  two  main  reasons  for  this: 

1)  I  was  course  director  and  only  instructor  for  two  successive  semesters  the  course  was 
taught. 

2)  Traditionally,  the  course  received  below  department  and  USAFA  averages  for  course 
critique  data  (this  occurred  the  first  time  I  taught  it  and  I  looked  for  ways  to  improve  the 
receptivity  of  the  course  material) 

The  first  semester  (Fall  '97)  was  a  traditional  lecture-based  and  exam  type  format.  The 
following  year  (Fall  '98)  was  taught  using  the  quiz-first  method.  Following  mid-course 
critique  data  that  was  remarkably  positive  compared  to  the  previous  time  I  taught  it,  I  was 
encouraged  to  administer  a  subjective  questionnaire  to  the  students  regarding  student 
receptivity  to  the  format. 

Methods 

Subjective  questionnaires  were  administered  at  midterm  and  at  the  end  of  the  semester. 

An  analysis  of  end-of-course  critique  data  from  fall  '97  and  fall  '98  was  conducted  as  well  as  a 
comparison  between  final  exam  scores  of  the  two  semesters. 
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For  the  subjective  questionnaire,  a  7-point  scale  was  used  (7  indicated  "highly  agree",  1 
indicated  "highly  disagree).  Questions  1-13  addressed  feedback  for  the  instructor.  Questions 
14-23  addressed  feedback  about  the  course.  The  first  three  question  were: 

1 .  I  preferred  this  method  of  assessment  to  the  traditional  GR  method. 

2.  I  retained  information  better  with  the  quiz-first  method  compared  to  the  traditional  GR 
method. 

3.  I  feel  that  I  could  have  taken  the  quizzes  without  having  studied  for  them  and  done 
fine. 

Results 

The  results  for  the  first  three  questions  are  indicated  in  Figme  1.  All  responses  were 
significantly  different  from  neutral,  and  there  was  no  difference  between  midterm  and  the  end 
of  the  semester. 


Preference  for  "Quiz-First" 


Question  # 

Figure  1.  Subjective  questionnaire  results. 


Figure  2  illustrates  the  improved  overall  acceptance  of  the  course  based  upon  end  of 
course  critiques.  Note  the  improvements  in  every  category. 
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Question  # 

Figure  2.  End  of  course  critique  data. 

The  final  analysis  was  a  comparison  of  final  exam  scores  between  the  two  semesters. 

There  was  a  significant  difference  in  the  scores,  with  the  traditional  method  of  instruction 
obtaining  a  slightly  higher  average:  75.5%  (fall  1997)  vs.  72.6%  (fall  1998),  t(85)  =  2.058,  p 
=  0.043. 

Discussion 

Independent  of  the  subjective  analyses,  the  results  of  the  final  exam  suggested  that  the 
efficacy  of  this  teaching  format  should  be  questioned.  However,  there  is  a  plausible 
explanation  for  the  difference  in  final  exam  scores.  The  final  exams  were  not  exactly  the 
same.  Roughly  75%  of  the  1998  exam  was  the  same  as  the  1997  exam.  Unfortunately,  due  to 
the  lack  of  primary  data  available  from  fall  1997,  the  similar  questions  could  not  be  accurately 
identified  or  separated  for  analysis.  Additionally,  almost  40%  of  the  questions  for  the  final 
were  given  to  fall  1997  students,  prior  to  the  exam.  This  was  not  the  case  for  the  fall  1998 
students.  Ostensibly,  they  could  have  come  to  the  final  exam  better  prepared  to  answer 
specific  questions  correctly.  Although  I  was  unable  to  asses  this  speculation  quantitatively,  a 
less  than  3%  difference  in  final  exam  scores  could  be  viewed  as  operationally  insignificant. 

Anecdotally,  there  are  several  reasons  that  this  appeared  to  be  an  effective  format  for 
enhancing  student  learning: 

1)  Students  were  put  "in  charge"  of  their  own  learning.  They  could  no  longer  come  to  class 
unprepared  and  expect  to  be  "spoon-fed"  the  material. 

2)  Students  appeared  to  be  more  alert  and  attentive  as  a  whole 

3)  Questions  that  students  asked  prior  to  qui22;es  and  during  lectures  were  more  intelligent 
and  inquisitive 

On  a  personal  level,  there  were  many  positive  aspects  of  this  methodology  for  the 
instructor.  The  format  allowed  me  the  flexibility  to  digress  from  the  syllabus  without  fear  of 
the  students  missing  important  material.  If  during  a  lecture  the  conversation  drifted  from  the 
lesson  topic  to  something  else,  I  was  comfortable  with  the  fact  that  the  students  had  seen  the 
material  before.  If  I  didn't  cover  it  in  the  lecture,  all  was  not  lost.  They  had  read  the  material  ■ 
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and  been  tested  on  it  already.  This  often  gave  me  the  opportunity  to  discuss  "real  life"  cadet 
wing  and  Air  Force  issues  with  them. 
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Encouraging  Active  Student  Learning;  Making  Investing  Relevant  and  Real 

David  R.  King,  M.S.,  Captain,  USAF* 

Department  of  Management 

Abstract:  Making  academic  material  relevant  and  real  is  important.  This  is  particularly  true  for 
personal  finance,  because  lessons  from  the  financial  "School  of  Hard  Knocks"  can  impact  a  person  for 
years.  A  pre  and  post-test  study  was  used  to  examine  the  impact  of  studying  and  practicing  investing 
as  part  of  the  Academy's  core  management  class. 

It  does  not  matter  how  many  times  you  were  told  not  to  touch  a  hot  stove  growing  up; 
you  only  remember  the  result  when  you  touched  it  yourself  the  first  time.  People  remember 
and  apply  the  lessons  they  experience  firom  the  "School  of  Hard  Knocks"  more  than  the 
information  or  advice  they  receive  from  other  sources.  The  challenge  for  educators  is  to  make 
academic  material  relevant  and  real  so  that  the  lessons  from  a  more  forgiving  academic 
environment  last  as  long  as  the  lessons  from  the  "School  of  Hard  Knocks." 

This  is  especially  important  in  the  realm  of  personal  finance  and  investing  because 
mistakes  in  this  area  can  impact  individuals  for  years.  For  example,  a  survey  by  Nellie  Mae, 
the  student  financial  aid  organization,  found  the  average  credit  card  debt  of  undergraduates 
(between  ages  18  and  25)  was  $1,879.^  With  an  average  credit  card  interest  rate  around  18 
percent  and  paying  a  monthly  minimum  of  $30  (assuming  no  additional  charges  were  made), 
it  would  take  14.5  years  and  cost  $4,375  to  pay  off  this  debt.  The  opportunity  cost  of  not 
investing  early  is  just  cis  high  because  of  the  similar  affect  on  compound  growth  in 
investments.  Assuming  a  12  percent  annual  return,  a  22-year-old  who  puts  $2,000  toward  an 
IRA  for  only  six  years  will  end  up  with  more  money  at  age  65  than  someone  who  waits  until 
they  are  28  to  begin  contributing  to  an  IRA.  This  remains  true  even  if  the  28-year-old  never 
stops  contributing  to  the  IRA  through  age  65. 

At  the  Air  Force  Academy,  the  challenge  of  giving  cadets  the  personal  finance 
knowledge  and  experience  they  need  is  one  task  imdertaken  in  the  core  management  course. 
Management  210.  During  the  fall  of  1998,  Management  210  cadets  invested  $100,000  of 
'virtual'  dollars  in  stocks,  bonds,  and  mutual  funds  with  the  help  of  a  real  time  stock  market 
simulation  website.^  Initially,  many  cadets  reported  that  they  were  intimidated  by  the  project, 
but  it  did  not  take  them  long  to  identify  with  Iheir  investments.  Additionally,  classroom 
discussion  was  real  and  personal  because  it  often  related  to  course  material  to  investments 
they  had  made.  This  paper  reports  on  the  process  of  moving  cadets  from  intimidation  to 
interest,  and  finally  to  active  learning. 

Research  Questions 

The  primary  goal  of  this  study  was  to  determine  if  combining  course  material  with  a 
real  time  experiential  exercise  would  alter  cadet  intentions  and  behavior  to  invest.  Several 
research  questions  were  developed  to  see  if  the  goal  was  achieved.  The  six  questions  I  set  out 
to  answer  included: 

•  What  is  the  baseline  financial  experience  of  cadets? 


'  Special  thanks  to  Dr.  Chirsty  Strbiak  for  her  mentorship  on  this  study. 

^  Greene,  Robert.  "Study  Finds  Most  College  Students  Use  Credit  Wisely,"  Denver  Post.  25  Jun  98,  p.  3C. 
^http:\\library.advanced.org\10326\market_simulation\index.html 
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•  What  are  the  financial  habits  and  intentions  of  cadets  prior  to  the  finance  lessons 
compared  to  afterward? 

•  How  confident  are  cadets  on  financial  matters  and  does  their  confidence  change 
over  the  course  of  the  semester? 

•  How  comfortable  are  cadets  v^th  managing  their  ovra  investments  and  does  it 
change  over  the  semester? 

•  What  is  the  knowledge  level  of  cadets  prior  to  the  finance  lessons  compared  to 
afterward? 

•  What  level  of  risk  tolerance  do  cadets  have  and  does  it  change  over  the  semester? 

Methods 

A  pre  and  post-test  experimental  design  was  used  during  the  fall  of  1998  to  measure 
the  impact  of  the  finance  project  and  lessons.  The  personal  finance  lessons  were  presented  at 
the  beginning  and  end  of  the  semester  with  the  investment  project  running  between  the  "book 
end"  finance  lessons.  The  initial  finance  lessons  were  designed  to  introduce  terminology  and 
concepts  the  cadets  would  need  in  order  to  successfully  participate  in  the  stock  market 
simulation.  The  final  finance  lessons  were  designed  to  summarize  the  experience  of  the  stock 
market  simulation  and  reinforce  the  importance  of  personal  finance. 

Participants.  Participants  were  a  subset  of  the  fall  1998  Management  210  course. 
Cadets  were  selected  for  participating  in  the  research  based  on  their  assignment  to  the  author's 
classes.  Cadets  were  initially  randomly  assigned  to  all  sections  of  the  Introduction  to 
Management  course.  This  sample  represented  77  of  the  591  cadets  enrolled.  Participation  in 
the  study  was  voluntary.  A  survey  was  used  to  answer  the  research  questions. 

Measures.  The  questions  for  the  survey  included  both  questions  from  existing 
literature  and  questions  developed  for  this  study.  The  questions  are  presented  in  Appendix  A. 
Questions  on  the  pre  and  post-test  were  identical  and  designed  to  measure  cadets  investment 
experience,  habits,  confidence,  intentions,  knowledge,  and  demographic  information.  The 
number  of  subjects  for  the  pre-test  (n  =  77)  and  post-test  (n  =  69)  varied  because  of  voluntary 
participation  and  absences  for  the  day  the  pre  and  post-test  were  administered. 

Parents'  income  (PI)  was  a  self-reported  measure  of  the  total  income  by  their  parent's. 
The  scale  went  from  $35,000  to  more  than  $120,000  with  $20,000  between  selections  on  the 
scale.  Amount  invested  (AI)  is  a  single  item  self-reporting  the  amount  invested  in  stocks, 
bonds,  or  mutual  funds.  The  scale  included  responses  varying  from  not  having  anything 
invested  to  separately  identifying  amounts  invested.  Investing  comfort  (IC)  was  a  single  item 
measuring  a  self-reported  response  in  reaction  to  a  statement.  The  scale  of  the  response  went 
from  strongly  agree  to  strongly  disagree.  Finance  confidence  (FC)  was  a  single  item  measure. 
It  was  a  self-reported  response  to  a  statement  with  a  scale  from  strongly  agree  to  strongly 
disagree.  Friends  and  family  (FF)  is  a  self-reported,  single  item  measure  of  the  frequency 
cadets  talk  with  fiiends  and  family  about  investing.  The  scale  of  the  response  went  from 
strongly  agree  to  strongly  disagree. 

Television  news  (TV)  was  a  self-reported,  single  item  measure  of  the  frequency  cadets 
watched  television  news  about  investing  or  the  stock  market.  The  range  of  responses  was 
from  at  least  once  a  day  to  never.  Newspaper  news  (NN)  was  a  self-reported,  single  item 
measure  of  the  frequency  cadets  read  the  business  section  of  a  newspaper.  The  range  of  the 
response  was  "at  least  once  a  day"  to  "never."  Business  periodicals  (BP)  was  a  self-reported. 
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single  item  measure  of  the  jfrequency  cadets  read  other  business  periodicals.  The  range  of  the 
response  was  "at  least  once  a  day"  to  "never." 

Stock  intent  (SI)  was  a  self-reported,  single  item  measure  of  whether  cadets  intend  to 
invest  in  stocks  within  the  next  year.  The  range  of  the  response  was  "definitely"  to  "not  at 
all."  Bond  intent  (BI)  was  a  self-reported,  single  item  measure  of  whether  cadets  intend  to 
invest  in  bonds  within  the  next  year.  The  range  of  the  response  was  "definitely"  to  "not  at 
all."  Mutual  fund  intent  (MFI)  was  a  self-reported,  single  item  measure  of  whether  cadets 
intend  to  invest  in  stocks  within  the  next  year.  The  range  of  the  response  was  "definitely"  to 
"not  at  all." 

Credit  card  payments  (CCP)  was  a  self-reported,  single  item  measure  of  cadet  habits 
in  paying  off  their  credit  card  debt.  The  range  of  the  response  was  from  not  having  a  credit 
card  to  paying  off  the  balance  each  month.  Knowledge  (K)  was  a  multiple  item  measure  of 
cadet  knowledge  about  investing  topics  and  the  score  was  based  on  the  number  of  correct 
responses.  Risk  (R)  was  a  multiple  item  measure  of  risk  based  on  responses  to  different 
scenarios.  Different  responses  were  weighted  as  involving  varying  levels  of  risk,  and  the  risk 
weighting  for  each  response  was  summed  to  create  a  score  along  a  range. 

The  final  stage  of  the  research  involved  analyzing  the  data.  Correlation 'analysis  was 
used  to  determine  how  closely  related  the  pre-test  measures  were  to  each  other.  It  was  also 
used  to  determine  how  closely  related  post-test  measures  were  to  each  other.  The  pre-  and 
post-test  data  were  analyzed  for  significant  differences  between  the  means  using  a  t-test  for 
two  independent  samples  (Sheskin,  1997).  Ideally,  a  t-test  for  two  dependent  samples  would 
be  the  most  appropriate,  but  due  to  anonymity  concerns  data  were  not  collected  in  matched 
pairs.  Additionally,  due  to  absences  and  voluntary  participation  sample  sizes  were  not 
identical. 

Results 

The  study  had  several  significant  findings.  The  findings  are  organized  by  reporting 
correlation  among  questions  on  the  pre  and  post-test.  Then  results  for  which  there  were 
significant  difference  between  the  means  on  the  pre  and  post-test  are  described. 

Pre-Test.  Knowledge  was  correlated  with  several  factors.  There  was  a  significant 
relationship  between  knowledge  and  credit  card  payments  (r  =  .388; p  <  .01),  talking  with 
friends  and  family  {r  =  -.317;  p  <  .05),  investing  comfort  (r  =  -.454;  p  <  .01),  risk  (r  =  .309;  p 

<  .01),  and  amount  invested  (r  =  .301;  p  <  .01).  Cadets  with  more  knowledge  about  investing 
were  more  likely  to  pay  off  their  credit  card  debt.  Cadets  with  more  knowledge  were  also  to 
have  higher  comfort  in  managing  their  own  investments  and  take  on  more  ris£  Finally, 
cadets  who  had  more  knowledge  were  more  likely  to  already  have  money  invested. 

Additionally,  finance  confidence  was  correlated  with  investing  comfort  (r  =  .405;  p 

<  .01).  The  more  confident  a  person  was  in  managing  their  day  to  day  finances  the  more 
comfortable  they  were  with  managing  their  own  investments.  There  was  also  a  significant 
relationship  between  talking  with  friends  and  family  and  a  cadet's  investing  comfort  (r  = 
.517;  p  <  .01),  and  amount  invested  (r  =  -.377; p  <  .01).  The  more  a  cadet  talked  with  friends 
and  family  about  investing  the  more  likely  they  were  to  be  comfortable  with  investing  and  to 
have  money  invested. 

The  frequency  with  which  a  person  would  look  at  one  news  source  (TV,  newspaper,  or 
periodical)  was  closely  correlated  with  whether  they  looked  at  other  sources.  There  was  a 
significant  relationship  between  watching  television  news  and  newspaper  reading  (r  =  -.397;  p 
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<  .01),  and  reading  business  periodicals  (r  =  .517;p  <  .01).  This  meant  that  a  cadet  that 
looked  at  one  source  of  investing  or  business  news  was  more  likely  to  look  at  other  sources  of 
news. 

Post-Test.  Knowledge  was  correlated  with  several  factors.  As  in  the  pre-test,  there 
was  a  significant  relationship  between  knowledge  and  investing  comfort  (r  =  -.336;  p  <  .01), 
and  talking  with  friends  and  family  (r  =  .260;  p  <  .051).  The  higher  knowledge  scores  related 
to  higher  investing  comfort  and  frequency  of  talking  about  investing  with  friends  and  family 
For  the  post-test  only,  knowledge  was  correlated  with  finance  confidence  (r  =  -346;  p  <  .01), 
newspaper  reading  (r  =  .368;  p  <  .01),  and  business  periodicals  (r  =  -.248;  p  <  .05).  Cadets 
demonstrating  higher  knowledge  were  more  likely  to  look  at  newspapers  and  business 
periodicals. 

As  in  the  pre-test,  the  relationship  between  finance  confidence  and  investing  confidence 
was  significant  (r  =  .552;  p  <  .01).  On  the  post-test,  finance  confidence  had  a  significant 
relationship  with  fiiends  and  family  (r  =  .361;  p  <  .01),  newspaper  news  (r  =  .351;p  <  .01), 
and  credit  card  payments  (r  =  -.365;  p  <  .01).  Cadets  with  higher  finance  confidence  were 
more  likely  to  talk  and  read  about  investing.  Additionally,  cadets  with  higher  finance 
confidence  were  more  likely  to  pay  off  credit  card  debt  each  month. 

Unlike  the  pre-test,  the  frequency  with  which  a  person  would  look  at  either  TV  or 
periodicals  was  closely  correlated  with  what  investment  they  intended  to  make.  There  was  a 
significant  relationship  between  watching  television  business  news  and  stock  intent  (r  =  .427; 
p  <  .01).  There  was  also  a  significant  relationship  between  reading  business  periodicals  and 
stock  intent  (r  =  .467;  p  <  .01).  Cadets  reading  newspaper  news  or  business  periodicals  were 
more  likely  to  have  intentions  of  investing  in  stocks.  There  was  also  a  significant  relationship 
between  reading  business  periodicals  and  mutual  fund  intent  (r  =  .256;  p  <  .05).  Cadets 
reading  business  periodicals  were  more  likely  to  have  intentions  of  investing  in  mutual  funds. 

Tests  of  Means.  The  question,  "How  comfortable  are  cadets  with  managing  their  own 
investments  and  does  it  change  over  the  semester?"  was  tested.  There  was  a  significant 
change  in  investing  comfort  (p  <  .001)  from  the  beginning  of  the  semester  (mean  =  3.06)  and 
the  end  of  the  semester  (mean  =  2.07).  Cadets  became  more  comfortable  handling  their  own 
investments.  The  question,  "How  confident  are  cadets  on  financial  maters  and  does  their 
confidence  change  over  the  course  of  the  semester?"  was  also  tested.  The  change  in 
confidence  for  handling  day  to  day  finances  was  also  significant  {p  <  .05),  pre-test  (mean  = 
2.00),  and  post-test  (mean  =  1.68). 

Next  the  question,  "What  is  the  knowledge  level  of  cadets  prior  to  the  finance  lessons 
compared  to  afterward?"  was  tested.  There  was  a  significant  change  in  investing  knowledge 
(p  <  .001)  from  the  beginning  of  the  semester  (mean  =  2.38)  and  the  end  of  the  semester 
(mean  =  5.20).  Cadets  became  more  knowledgeable  about  investing  over  the  course  of  the 
semester. 

Finally,  results  showed  that  cadets  changed  their  habits  over  the  course  of  the 
semester.  The  first  habit  tested  was  talking  to  friends  and  family.  There  was  a  significant 
change  in  the  amoimt  that  cadets  talked  with  fnends  and  family  about  investing  (p  <  .001) 
from  the  beginning  of  the  semester  (mean  =  2.52)  to  the  end  of  the  semester  (mean  =  1 .80). 
Cadets  were  more  likely  to  talk  about  investing  after  taking  Management  210.  Next, 
investing  habits  were  examined.  The  different  investment  vehicles  (stock,  bond,  and  mutual 
fund)  were  tested  for  significant  change.  No  significant  changes  occurred  from  the  pre-test  to 
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the  post-test  on  stock  or  bond  intent  {p  <  .2).  However,  there  was  a  significant  change  for 
mutual  fund  intent  (p  <  .05).  Cadets  were  more  likely  to  invest  in  mutual  funds  after  the 
course  (x  =  1 .46)  than  before  (x  =  1 .73).  This  showed  that  cadets  learned  that  the  best 
investment  based  on  diversification,  professional  management,  and  long-term  returns  was  a 
mutual  fund,  and  adjusted  their  future  investing  plans  accordingly. 

Discussion 

In  general,  I  set  out  to  answer  six  questions.  The  original  questions  with  the  answers 
summarized  follow.  The  first  research  question  involved  determining  the  baseline  financial 
experience  of  cadets.  Results  of  the  pre-test  showed  that  cadets  investing  experience  was 
bimodal.  Thirty-seven  of  the  77  cadets  had  not  invested  in  stocks,  bonds,  or  mutual  funds  at 
the  beginning  of  the  project,  while  twenty-five  had  over  $5,000  invested  in  financial 
instruments.  The  remaining  fifteen  cadets  had  less  than  $5,000  invested.  Even  though  48% 
of  the  cadets  had  money  invested,  in-class  interactions  showed  they  were,  apprehensive  of  the 
finance  project  at  the  beginning  of  the  semester.  The  reason  for  this  was  that  the  finance 
project  was  the  first  time  most  of  them  would  make  actual  decisions  about  companies  or  funds 
to  select  as  investments. 

The  second  research  question  involved  determining  changes  in  the  financial  habits  and 
intentions  of  cadets  before  and  after  the  finance  lessons.  There  was  a  significant  change  in  the 
amount  of  discussion  about  investing  that  cadets  had  with  friends  and  family.  This  could  be 
explained  by  the  fact  that  investing  was  a  requirement  of  the  course,  except  for  qualitative 
comments  from  cadets  that  once  they  started  investing  they  found  it  interesting  and  wanted  to 
learn  more.  Exposing  cadets  to  finance  in  Management  210  removed  the  fear  and  mystery  of 
investing,  and  increased  the  likelihood  that  they  would  talk  about  financial  matters.  The  only 
significant  change  in  cadet  investment  intentions  was  that  cadets  were  significantly  more 
likely  to  invest  in  mutual  funds  after  the  finance  lessons. 

The  third  and  fourth  research  questions  involved  determining  changes  in  cadet 
confidence  on  financial  matters  and  comfort  with  investing  before  and  after  the  finance 
lessons.  In  general  cadets  were  confident  in  their  ability  to  handle  their  day  to  day  finances 
throughout  the  semester.  However,  cadet  comfort  with  handling  their  investments  increased 
significantly. 

The  fifth  research  question  involved  determining  changes  in  the  knowledge  level  of 
cadets  before  and  after  the  finance  lessons.  On  the  average,  cadet  knowledge  on  investing, 
like  the  general  publics,  was  low.  The  average  knowledge  level  of  cadets  after  the  finance 
lessons  increased  from  an  average  correct  of  2.37  out  of  9  questions  to  5.2  correct  or  57.8 
percent.  Based  upon  participation  in  class,  cadet  investing  knowledge  was  taken  from  below 
average  to  an  approximately  average  level  compared  to  mutual  fund  investors.  This 
observation  was  based  upon  a  comparison  of  cadet  results  to  the  results  of  1,555  mutual  fund 
investors  on  a  twenty-question  survey  by  Vanguard  for  which  the  average  score  was  51%."^ 
Thus,  cadet  knowledge  increased  to  a  level  similar  to  general  mutual  fund  investors. 

The  final  research  question  involved  determining  changes  in  cadet  investment  risk 
tolerance.  On  the  average,  cadets  were  moderate  to  aggressive  risk  takers  throughout  the 
semester.  In  the  pre-test,  there  was  a  significant  correlation  between  risk  and  finance 
confidence  (r  =  -.319;  j?  <  .01),  and  knowledge  (r  =  .309;  p  <  .01).  This  meant  that  more 
knowledgeable  and  confident  cadets  were  comfortable  with  more  risk.  However,  correlations 

* - ,  "Mutual  Fund  Literacy  Test,"  Vanguard  Marketing  pamphlet,  1998,  p.  1. 
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among  the  multiple  risk  questions  had  a  low  Cronbach  alpha,  suggesting  that  the  questions 
may  not  have  measured  the  same  factors  of  risk. 

,  The  study  had  a  number  of  limitations.  First,  generalizing  the  results  of  this  study  to 

college  students  in  general  may  be  difficult.  As  noted  earlier,  cadet  responses  on  the  pre-  and 
post-test  were  not  paired.  Additionally,  the  risk  measure  questions  may  not  have  measured 
the  same  factors  of  risk.  Other  issues  dealt  with  the  stock  project.  During  the  semester, 
problems  with  the  Internet  server  used  by  cadets  detracted  from  stock  market  simulation 
because  cadets  often  were  unable  to  access  it.  Also,  the  stock  market  experienced  a  correction 
at  the  beginning  of  the  project.  Recommendations  for  future  studies  include:  using  a  control 
group,  collecting  data  in  matched  pairs,  and  correlating  both  initial  experience  and  confidence 
with  the  results  of  the  investment  game. 

Summary 

This  study  highlighted  a  number  of  interesting  issues.  First,  cadets  that  already  had 
money  invested  in  financial  instruments  were  more  likely  to  pay  off  their  credit  card  in  full 
each  month  and  talk  with  friends  and  family  about  investing.  Additionally,  cadets  with  prior 
investing  experience  tended  to  know  more  about  investing  from  the  very  beginning.  Finally, 
cadet  knowledge  on  investing  in  general  was  low.  This  is  a  concern  because  poor  financial 
decisions  now  can  have  long  lasting  impacts.  The  majority  of  participants  were  either  juniors 
or  seniors  and  avoiding  financial  problems  is  part  of  a  successfol  military  career. 

Fortunately,  investing  knowledge  increased  to  at  least  the  average  level  of  mutual  fund 
investors  in  general  by  the  end  of  Management  210.  Another  positive  indication  on  the  post¬ 
test  was  that  knowledge  was  correlated  with  comfort  in  managing  their  investments,  or 
investing  comfort  was  supported  by  better  knowledge  about  investing.  Additionally,  the  only 
significant  change  in  investing  intentions  involved  mutual  funds.  This  can  be  viewed 
positively  because  mutual  funds  offer  lower  risk  because  of  diversification  than  direct 
investment  in  stocks. 

In  conclusion,  active  involvement  and  presentation  of  course  material  in  Management 
210  changed  cadets'  knowledge,  behavior,  and  investing  intentions.  Because  the  finance 
lessons  and  project  dealt  with  "virtual"  dollars,  negative  consequences  could  be  experienced 
without  long-term  financial  harm.  Learning  similar  lessons  from  the  "School  of  Hard 
Knocks"  with  their  hard-earned  money  would  be  less  forgiving.  Ideally,  each  cadet's 
investing  experience  in  Management  210  gave  them  relevant  experiences  that  should  help 
them  avoid  financial  lessons  from  the  "School  of  Hard  Knocks."  Several  cadets  also  showed 
interest  in  continuing  to  learn  about  investing,  reinforcing  the  observation  that  cadet  behavior 
and  intentions  were  modified.  Finally,  when  cadets  who  experienced  the  Management  210 
finance  lessons  become  officers,  they  should  be  more  likely  to  make  better  investment 
decisions. 

Reference 
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Appendix  A:  Pre/Post-Test 


Section  1:  Background  Questions 

1)  What  amount  do  you  currently  have  invested  in  stocks,  bonds,  or  mutual  funds? 

a.  Do  not  have  investments  in  stocks,  bonds,  or  mutual  funds 

b.  Have  investments  in  stocks,  bonds,  or  mutual  funds  but  do  not  know  value  of  investment 

c.  $1  to  $1000 

d.  $1001  to  $5000 

e.  Over  $5000 

2)  What  is  your  reaction  to  the  following  statement:  "I  am  comfortable  with  managing  my  own 
investments  for  retirement  and  other  goals." 

a.  Strongly  agree 

b.  Agree 

c.  Neither  Agree/Disagree 

d.  Disagree 

e.  Strongly  disagree 

3)  What  is  your  reaction  to  the  following  statement:  "I  am  confident  in  my  ability  to  manage  my  day 
to  day  personal  finances." 

a.  Strongly  agree 

b.  Agree 

c.  Neither  Agree/Disagree 

d.  Disagree 

e.  Strongly  disagree 

4)  What  is  your  reaction  to  the  following  statement:  "I  talk  about  investing  with  friends  and  family." 

a.  Strongly  agree 

b.  Agree 

c.  Neither  Agree/Disagree 

d.  Disagree 

e.  Strongly  disagree 

5)  How  often  do  you  do  watch  TV  news  talking  about  investing  or  the  stock  market? 

a.  At  least  once  a  day 

b.  More  than  once  a  week 

c.  About  once  a  week 

d.  Once  a  month  or  less 

e.  Never 

6)  How  often  do  you  read  the  business  section  of  a  newspaper? 

a.  At  least  once  a  day 

b.  More  than  once  a  week 

c.  About  once  a  week 

d.  Once  a  month  or  less 

e.  Never 

7)  How  often  do  you  read  other  business-related  periodicals? 

a.  At  least  once  a  day 

b.  More  than  once  a  week 

c.  About  once  a  week 

d.  Once  a  month  or  less 

e.  Never 

8)  Do  you  intend  to  invest  money  in  stocks  in  the  next  year? 

a.  Definitely 

b.  Maybe 
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c.  Not  at  all 

9)  Do  you  intend  to  invest  money  in  bonds  in  the  next  year? 

a.  Definitely 

b.  Maybe 

c.  Not  at  all 

10)  Do  you  intend  to  invest  money  in  mutual  funds  in  the  next  year? 

a.  Definitely 

b.  Maybe 

c.  Not  at  all 

11)  When  you  make  your  monthly  credit  card  payments,  do  you? 

a.  Do  not  have  a  credit  card 

b.  Make  the  minimum  payment 

c.  Pay  more  than  the  minimum  payment,  but  less  than  half  the  complete  balance 

d.  Pay  more  than  the  minimum  payment  and  pay  more  than  half  the  complete  balance,  but 
less  than  the  complete  balance 

e.  Pay  off  the  complete  balance  every  month 

12)  You  work  for  a  small,  but  thriving,  privately  held  electronics  company.  The  company  is  raising 
money  by  selling  its  stock  to  employees.  Management  plans  to  take  the  company  public,  but  not  for  4 
more  years.  If  you  buy  the  stock  you  will  not  be  able  to  sell  it  until  the  shares  are  traded  publicly.  In 
the  meantime,  the  stock  will  pay  no  dividends.  But  the  shares  could  trade  for  10  to  20  times  more  than 
what  you  paid  for  them  when  the  company  goes  public.  How  much  would  you  invest  in  the  company? 

a.  Nothing 

b.  One  month's  salary 

c.  Three  month's  salaiy 

d.  Six  month's  salary 

13)  You  are  on  a  TV  game  show  and  can  choose  one  of  the  following.  Which  one  would  you  choose? 

a.  $1,000  cash 

b.  50  percent  chance  of  winning  $4,000 

c.  20  percent  chance  of  winning  $  1 0,000 

d.  5  percent  chance  of  winning  $100,000 

14)  You  have  lost  $500  at  the  blackjack  tables  in  Las  Vegas.  How  much  more  are  you  prepared  to 
lose  to  win  the  $500  back? 

a.  Nothing 

b.  $100 

c.  $250 

d.  $500 

e.  Over  $500 

15)  Your  investment  loses  15  percent  of  its  value  in  a  market  correction  a  month  after  you  buy  it. 
Assuming  that  none  of  the  fundamentals  have  changed,  do  you: 

a.  Sit  tight  and  wait  for  it  to  journey  back  up 

b.  Sell  it  and  rid  yourself  of  further  sleepless  nights  in  case  it  continues  to  decline 

c.  Buy  more— if  it  looked  good  at  the  original  price  it  looks  even  better  now 

16)  Have  you  participated  in  an  investing  or  stock  market  game  before? 

a.  Yes 

b.  No 

Section  2:  Topic  Questions 

17)  If  you  buy  a  $10,000  bond  paying  10  percent  interest,  what  happens  to  the  value  of  the  bond  when 
interest  rates  fall? 

a.  Do  not  know 
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b.  The  value  of  the  bond  does  not  change 

c.  The  value  of  the  bond  increases 

d.  The  value  of  the  bond  falls 

18)  Dividing  a  company's  share  price  by  a  summary  of  its  last  four  quarter's  earnings  is  a  method  of 
finding  the: 

a.  Do  not  know 

b.  PE  ratio 

c.  Dividend  yield 

d.  Current  ratio 

e.  Profit  margin 

19)  What  stock  market  average  represents  30  "Blue  Chip"  stocks  trading  on  the  NYSE? 

a.  Do  not  know 

b.  S&P500 

c.  Russell  2000 

d.  Dow  Jones  Industrial  Average 

e.  Wilshire  5000 

20)  When  a  company  increases  the  amount  of  stock  to  reduce  its  share  price  by  issuing  stockholders 
additional  shares,  this  is  called  a: 

a.  Do  not  know 

b.  Short  sale 

c.  Stock  split 

d.  Capital  gain 

e.  Goodwill 

21)  When  a  company  announces  a  corporate  takeover,  what  generally  happens  to  the  share  price  of  the 
company  being  acquired? 

a.  Do  not  know 

b.  Takeovers  do  not  impact  share  prices 

c.  Share  price  increases 

d.  Share  price  decreases 

22)  A  mutual  fund  that  tries  to  maintain  a  net  asset  value  of  $1  by  investing  in  short  term  securities  is 
called  a: 

a.  Do  not  know 

b.  Bond  fund 

c.  Balanced  fund 

d.  Money  market  fund 

e.  Equity  fund 

23)  What  type  of  investment  has  historically  provided  the  best  protection  against  inflation? 

a.  Do  not  know 

b.  Stocks 

c.  Bonds 

d.  Savings  accounts 

e.  Real  estate 

24)  If  two  mutual  funds  hold  the  same  securities,  but  one  has  higher  operating  expenses  than  the  other, 
what  is  the  impact  on  the  fund's  return? 

a.  Do  not  know 

b.  The  fund  with  the  higher  expenses  will  have  a  higher  return 

c.  The  fund  with  the  lower  expenses  will  have  a  higher  return 

d.  Operating  expenses  do  not  impact  fund  returns 

25)  In  general,  which  of  the  following  types  of  mutual  funds  is  likely  to  decline  the  most  when  the 
overall  stock  market  falls? 
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a.  Do  not  know 

b.  Aggressive  growth  fund 

c.  Growth  and  Income  fund 

d.  Bond  fund 

e.  Money  Market  fund 

26)  Who  was  President  of  the  United  States  during  the  1929  market  crash? 

a.  James  Garfield 

b.  Hubert  Hoover 

c.  Calvin  Coolidge 

d.  Teddy  Roosevelt 

Section  3:  Demographics 

27)  What  is  your  gender? 

^a.  Male 

b.  Female 

28)  What  class  are  you  in? 

a.  Firstie 

b.  2  degree 

c.  3  degree 

d.  Doolie 

29)  What  is  your  age? 

a.  25  or  26 

b.  23  or  24 

c.  21  or  22 

d.  19  or  20 

e.  17  or  18 

30)  What  is  your  parent's  income? 

a.  More  than  $  1 20,000 

b.  $75,000  to  $119,999 

c.  $55,000  to  $74,999 

d.  $35,000  to  $54,999 

e.  under  $35,000 
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Designing  an  Honors  Leadership  Seminar 

John  Micalizzi,  Ph.D.,  Lt  Colonel,  USAF  - 
Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  During  the  Spring  of  1998, 1  was  given  the  opportunity  to  create  a  unique  leadership  course 
for  a  section  of  firstclass  and  secondclass  behavioral  sciences  majors.  The  structure  of  the  course 
emphasized  student  involvement  in  developing  and  delivering  the  leadership  lessons.  Student  critique 
ratings  reflected  overall  student  satisfaction  with  this  new  approach. 

Background 

During  the  Spring  of  1998, 1  was  given  the  opportunity  to  create  a  unique  leadership 
experience  for  a  section  of  firstclass  and  secondclass  behavioral  sciences  majors  enrolled  in 
Behavioral  Sciences  310,  Leadership  Concepts  and  Applications.  As  the  only  academic 
course  in  leadership  offered  to  cadets  during  their  four  years  at  USAF  A;  Behavioral  Sciences 
310  had  evolved  over  the  years  to  be  primarily  a  survey  course  covering  a  variety  of  basic 
psychological  principles  and  theories  as  they  applied  to  the  concept  of  leadership.  Having 
taught  this  course  using  the  traditional  textbook/syllabus  approach  for  several  years,  I  was 
curious  to  try  a  new  approach  that  would  take  advantage  of  the  leadership  training  and 
experiences  students  have  outside  the  classroom  as  cadet  leaders  and  followers.  Since 
Academy  cadets  are  probably  more  familiar  with  general  leadership  issues  than  typical 
college  students  attending  the  same  kind  of  course.  I’ve  always  toyed  with  the  idea  of 
allowing  my  students  to  create  their  own  leadership  experience  in  the  classroom.  This  fantasy 
came  true  last  Spring  semester. 

The  First  Day 

Place  yourself  in  the  role  of  one  of  my  students  attending  the  first  day  of  class  who 
hears  me  say,  “In  this  course,  there  will  be  no  textbook,  syllabus,  assigned  readings  or 
periodic  quizzes/Graded  Reviews.  Everyone  in  this  class  will  be  responsible  for  working 
together  to  create  an  effective  learning  environment  for  studying  leadership,  beginning  with 
the  development  of  an  appropriate  syllabus  due  on  Lesson  10.  The  material  for  the  course  can 
come  from  the  30  or  so  leadership  references  that  I  will  make  available  to  you  or  any  other 
leadership  source  you  may  discover.  You  may  organize  yourselves  as  you  see  fit  to 
accomplish  the  assigned  tasks.  Are  there  any  questions?”  Student  reaction  ranged  from 
disbelief,  to  joy,  to  mistrust,  to  “why  me?”  One  objective  of  this  approach  was  to  provide  a 
forum  where  students  would  directly  participate  in  leadership/followership  interactions  while, 
at  the  same  time,  developing  a  plan  for  studying  these  phenomena. 

The  Plan 

While  student  choice  and  flexibility  were  the  major  themes  of  this  course,  there  was  a 
certain  amount  of  structure  imposed  to  focus  their  learning  experience.  First,  a  syllabus  was 
required  which,  among  other  things,  specified  the  content  and  sequencing  of  ten  student-led 
discussions  beginning  on  Lesson  12.  Working  in  pairs,  students  were  responsible  for 
selecting,  designing,  and  delivering  a  50-minute  lesson  on  a  leadership  topic  of  their  choice. 
During  the  lessons  leading  up  to  the  student  presentations,  I  would  construct  short  lectures, 
lead  discussion  groups,  and  conduct  various  classroom  leadership  assessment  exercises  to 
create  the  framework  for  future  leadership  discussions. 
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A  leadership  case  study  construction  exercise  was  scheduled  to  kick  off  the  second 
half  of  the  semester.  Cadets  worked  in  groups  of  four  to  create  richly  detailed  cadet-based 
leadership  scenarios,  including  a  recommended  resolution  of  the  issues;  that  could  be  used  to 
illustrate  the  concepts  discussed  in  class.  Each  group  would  then  be  handed  another  group’s 
scenario  and  asked  to  resolve  that  scenario,  first  as  individuals  outside  class  and,  later,  as 
groups  in  class. 

The  last  third  of  the  semester  was  reserved  for  cadets  to  prepare  and  carry  out  their 
final  projects.  The  projects  required  student  groups  to  consult  with  selected  base 
organizations  on  a  particular  leadership  issue.  Through  this  partnership,  the  cadet  consultants 
helped  their  organizations  define  the  relevant  leadership  issues,  develop  alternative  strategies 
to  deal  vrith  these  issues,  and  implement  the  selected  approach.  The  results  were  written  up  in 
a  project  paper  and  presented  to  the  rest  of  the  class  during  a  fifty-minute  briefing. 

Students  were  also  required  to  take  a  final  exam  which  consisted  of  3  essay  questions 
designed  to  integrate  student  learning  across  the  semester. 

In  addition,  my  students  participated  in  many  of  the  special  activities  associated  with 
the  general  course  including  an  “Airman”  panel,  a  “Noncommissioned  Officers”  panel,  and 
the  Leadership  Reaction  Course.  And  finally,  my  students  were  required  to  record  up  to  six 
personal  leadership  experiences  throughout  the  semester  using  the  Action-Observation- 
Reflection  (AOR)  model  of  experiential  learning. 

The  Results 

The  results  of  this  unique  classroom  experience  greatly  exceeded  my  expectations  as 
the  instructor.  In  general,  the  overall  amount  and  quality  of  leadership  discussions  were 
unmatched  by  any  class  I’ve  seen  in  my  ten  years  of  teaching  at  the  Academy. 

Several  student  products  were  evaluated,  in  part,  by  outside  instructors  with  the 
criteria  developed  by  students  in  consultation  with  myself  For  example,  the  class  syllabus 
was  graded  by  five  instructors  including  the  Course  Director  for  the  main  course.  All 
evaluations  were  A-  or  higher.  A  similar  process  was  used  for  evaluating  the  content  of  the 
case  studies  where  grades  ranged  from  B+  to  A.  In  addition,  other  instructors  would  report 
that  some  of  my  students  would  initiate  conversations  in  their  courses  concerning  the  exciting 
and  positive  experiences  that  were  going  on  in  my  class. 

Student  performance  was  higher  in  my  section  as  indicated  by  an  overall  section  GPA 
of  3.4  compared  to  a  course  GPA  of  3.2. 

Probably  the  most  powerful  evidence  for  the  effectiveness  of  this  course  on  student 
learning  came  from  the  student  critique  ratings  measured  at  the  end  of  the  semester.  Using  a 
six-point  scale,  the  mean  student  rating  for  310  Honors  across  the  ten  course  dimensions  was 
5.4  compared  to  4.7  for  the  rest  of  the  course.  In  particular.  Intellectual  Challenge  and 
Independent  Thought  dimension  was  5.5H/4.7,  while  Amoimt  Learned  was  5.6H/4.6. 

For  me  personally,  the  most  satisfying  result  occurred  whenever  students  from  this 
section  would  approach  me  afterward  to  ask  if  I  was  plaiming  to  teach  another  course  using 
the  same  approach  so  that  they  could  consider  trying  to  fit  it  into  their  already  tight  schedules. 
That  was  the  most  persuasive  evidence  I  needed  to  keep  pushing  for  greater  student  control  in 
the  classroom. 
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Distributed  Practice:  More  Bang  for  Your  Homework  Buck 

Marie  A.  Revak,  Ph.D.,  Major,  USAF 
Director  of  Faculty  Development 
Directorate  of  Education 

Abstract:  Homework  is  commonplace  in  math  classrooms,  yet  little  research  has  been  conducted  on 
the  differential  effectiveness  of  homework  for  students  with  vatying  aptitudes.  In  this  study, 
distributed  practice  homework  bolstered  the  achievement  of  low  achieving  college  math  students.  The 
sample  consisted  of  351  US  Air  Force  Academy  cadets  all  in  their  first  semester  of  college.  An 
algebra/trigonometry  placement  exam  measured  prior  mathematics  achievement  and  a  subset  of  25 
items  from  the  Math  Anxiety  Rating  Scale  measured  math  anxiety  (Alexander  &  Martray,  1989). 
Data  were  analyzed  using  hierarchical  multiple  regression.  Treatment  group  students  outscored 
control  group  students  on  4  of  the  6  achievement  measures  without  regard  for  prior  math  achievement 
or  math  anxiety  (alpha  =  .05). 

Homework  is  commonplace  in  college  mathematics  cotirses,  yet,  with  the  exception  of 
the  inconclusive  research  investigating  Saxon’s  incremental  continuous  review  method 
(Abrams,  1989;  Denson,  1989;  Gianniotes,  1989;  Johnson  &  Smith,  1987;  Klingele  &  Reed, 
1984;  Parker,  1990;  Reed,  1983;  Rentschler,  1995;  Roberts,  1994;  Saxon,  1982),  little 
research  has  been  conducted  on  the  content  or  quality  of  mathematics  homework  or  on 
homework’s  effect  on  achievement.  Other  than  a  small  study  conducted  by  Hirsch,  Kapoor, 
and  Laing  (1982,  1983;  N  =  52  first  semester  college  calculus  students),  there  is  a  lack  of 
research  investigating  the  differential  effectiveness  of  homework  for  students  with  varying 
aptitudes  (Austin,  1979;  Featherstone,  1985;  Hirsch  et  al.,  1982,  1983;  Kohler  &  Grouws, 
1992;  Peterson,  1971;  Suydam,  1985).  College  students  placed  into  precalculus  and  algebra 
courses  have  not  yet  mastered  the  fundamentals  of  algebra  required  to  succeed  in  calculus. 
Many  of  these  students  have  learned  algebra  as  a  set  of  rules  for  attacking  specific  types  of 
problems.  Homework  problems  in  algebra  courses  usually  consist  of  a  set  of  problems  related 
to  the  most  recent  problem  type,  that  is,  massed  practice.  With  massed  practice,  students  do 
not  practice  learning  to  differentiate  between  problem  types.  Yet,  success  in  calculus  requires 
students  to  determine  when  and  where  to  use  a  variety  of  algebraic  techniques. 

By  assigning  homework  problems  related  only  to  the  most  current  course  topics, 
mathematics  educators  have  ignored  the  findings  of  cognitive  psychology  research 
recommending  spaced  over  massed  practice  (Dempster,  1988, 1989;  Reynolds  &  Glaser, 
1964).  Distributed  practice  is  based  on  the  aspect  of  information  processing  learning  theory 
known  as  the  spacing  effect. 

The  spacing  effect  is  the  phenomenon  in  which  “for  a  given  amount  of  study  time, 
spaced  presentations  yield  substantially  better  learning  than  do  massed  presentations” 
(Dempster,  1988,  p.  627).  The  spacing  effect  has  a  long  history  in  cognitive  psychology  and 
education  research  and  is  also  referred  to  as  distributed  practice,  continuous  review,  and 
spaced  review  (Cuddy  &  Jacoby,  1982;  Dempster,  1988;  Krug,  Davis,  &  Glover,  1990; 
Reynolds  &  Glaser,  1964;  Toppino  &  Gracen,  1985;  Underwood,  1961).  According  to 
Dempster  (1988),  although  distributed  practice  is  “one  of  the  most  remarkable  phenomena  to 
emerge  from  laboratory  research”  (p.  627),  there  is  little  evidence  that  its  potential  has  been 
realized  in  applied  settings. 

Research  on  distributed  practice  is  situated  in  information  processing  theory  (Ausubel, 
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1966).  For  over  25  years,  cognitive  psychology  research  has  documented  the  benefit  of 
spaced  practice  (Cuddy  &  Jacoby,  1982;  Krug  et  al.,  1990;  Melton,  1970;  Modigliani,  1976; 
Rea  &  Modigliani,  1985;  Toppino  &  Gracen,  1985;  Thorndike,  1971;  Underwood,  1961). 

The  most  typical  finding  of  this  research  was  that  as  spacing  increased,  retention  also 
increased.  However,  most  research  pertaining  to  the  spacing  effect  has  investigated  the 
learning  of  simple  word  or  number  lists  with  time  lags  measured  in  seconds.  Although  the 
spacing  effect  is  “one  of  the  most  robust  phenomena  discovered  in  memory  research”  (Rea  & 
Modigliani,  1985,  p.  1 1),  results  from  cognitive  psychology  experiments  do  not  necessarily 
transfer  to  complex  learning  tasks  with  longer  spacings  between  reviews  (Reynolds  &  Glaser, 
1964).  According  to  Dempster  (1988),  studies  conducted  from  a  basic  research  perspective 
and  those  conducted  from  an  applied  perspective  frame  two  distinct  research  strands. 

According  to  Cronbach  and  Snow,  “an  interaction  is  said  to  be  present  when  a 
situation  has  one  effect  on  one  kind  of  person  and  a  different  effect  on  another”  (1977,  p.  3). 
Salomon  (1972)  described  aptitude-treatment  interaction  (ATI)  research  as  accomplishing  two 
functions:  improving  instruction  and  advancing  instructional  theory.  Salomon’s 
compensatory  ATI  model  proposed  that  ATI  treatments  could  be  developed  to  interact  with 
aptitudes  by  circumventing  their  debilitating  effects  without  trying  to  improve  them.  Snow 
(1977)  advocated  the  use  of  some  measure  of  general  ability  in  all  instructional  research. 
Whenever  affective  traits  are  considered,  researchers  should  expect  that  the  regression  of  the 
trait  will  vary  with  ability.  Cronbach  and  Snow  (1977)  assert  that  the  anxiety  experienced  by 
an  individual  depends  on  the  difficulty  he  or  she  has  with  a  task.  Task  difficulty  depends  on 
an  individual’s  ability  and  the  characteristics  of  the  task.  Therefore,  a  complex  task  is  more 
likely  to  create  anxiety  in  persons  of  low  ability  than  in  more  able  persons  (Cronbach  & 
Snow). 

From  an  ATI  standpoint,  Tobias  (1976, 1989)  hypothesized  that  students  with  lower 
prior  achievement  require  more  instructional  support,  and  conversely,  that  as  the  level  of  prior 
achievement  increases,  less  instructional  support  may  be  required.  In  their  review  of  ATI 
research  in  science  education,  Koran  and  Koran  (1984)  referred  to  task  organization  as  a 
manipulation  likely  to  have  an  obvious  effect  on  learning  and  a  clear  implication  for  ATI 
research.  That  is,  material  that  is  well  organized  should  result  in  better  achievement  for  high 
anxiety  students  (Koran  &  Koran,  1984).  Similarly,  Tobias  (1989)  and  Bessant  (1995) 
recommended  clearly  structured  instruction  as  beneficial  to  highly  anxious  students. 
According  to  Sieber,  O’Neill,  and  Tobias  (1977),  students  high  in  anxiety  may  also  benefit 
from  opportunities  for  repetition  of  selected  parts  of  the  content. 

In  this  study,  the  spacing  principle  was  applied  to  Precalculus  homework  assignments 
(Hirsch  et  al.,  1982, 1983;  Peterson,  1971).  The  purpose  of  the  study  was  to  explore 
distributed  practice  homework  assignments  as  one  way  to  provide  the  instructional  support 
and  task  organization  necessary  to  increase  the  mathematics  achievement  of  students  with  low 
prior  mathematics  achievement,  high  levels  of  mathematics  anxiety,  or  both. 

Three  research  questions  were  established: 

1 .  Will  distributed  practice  homework  assignments  have  a  positive  effect  on  Precalculus 
achievement? 

2.  Will  distributed  practice  homework  assignments  have  a  greater  positive  effect  on 
Precalculus  achievement  than  traditional  homework  assignments  for  students  with  low 
prior  mathematics  achievement? 

3.  Will  distributed  practice  homework  assignments  have  a  greater  positive  effect  on 
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Precalculvis  achievement  than  traditional  homework  assignments  for  students  with 

high  mathematics  anxiety? 

Methods 

Participants.  The  sample  for  the  study  consisted  of  all  375  United  States  Air  Force 
Academy  (USAFA)  cadets  enrolled  in  Precalculus  during  the  1995  fall  semester.  Enrollment 
in  Precalculus  was  based  on  placement  exam  scores.  Students  scoring  less  than  50%  on  the 
Algebra/Trigonometry  placement  exam  were  placed  into  Precalculus.  The  sample  represented 
about  28%  of  the  first  year  students.  Of  the  remaining  first  year  students,  519  (about  39%) 
were  placed  into  Calculus  1, 344  (about  26%)  were  placed  into  Calculus  II,  and  103  (about 
8%)  were  placed  into  Calculus  III.  All  USAFA  students  are  required  to  complete  a  sequence 
of  core  courses  which  includes  at  least  two  semesters  of  Calculus. 

Natural  attrition  of  students  resulted  in  a  changing  sample  size  during  the  semester.  At 
the  time  of  the  first  exam,  351  of  the  original  375  cadets  enrolled  in  Precalculus  remained. 
Enrollment  was  341  at  the  time  of  the  second  exam,  338  at  the  time  of  the  third  exam,  and 
333  at  the  end  of  the  semester. 

The  USAFA  has  high  admission  standards.  To  qxxalify  for  admission,  students  must 
have  good  grades  and  athletic  and  leadership  experience  (Air  Force  Academy  Admissions 
Office,  1995).  In  addition,  students  must  be  unmarried,  without  dependents,  and  between  the 
ages  of  17  and  21  (Air  Force  Academy  Admissions  Office).  The  mean  Scholastic 
Achievement  Test  (SAT)  math  achievement  score  for  incoming  Air  Force  Academy  students 
was  660  (recomputed  to  reflect  the  1995  recentering  of  the  SAT)  and  the  mean  for  the  math 
portion  of  the  American  College  Test  (ACT)  for  incoming  students  was  29.3  (B.  A.  Branum, 
personal  communication,  September  6, 1995).  The  average  high  school  grade-point  average 
for  incoming  cadets  was  3.85  (B.  A.  Branum)  and  89%  of  entering  cadets  ranked  in  the  top 
fifth  of  their  high  school  class  (Air  Force  Academy  Admissions  Office). 

The  USAFA  class  of  1999  consisted  of  1367  students,  1353  fi-om  the  United  States 
and  14  from  13  foreign  countries  (Lockhart,  1995).  Included  were  238  minority  members 
(17%)  and  219  women  (16%).  Of  the  United  States  students,  1086  (82%)  were  White,  56 
(4%)  were  Black,  85  (6%)  were  Hispanic,  72  (6%)  were  Asian  American,  and  19  (1%)  were 
Native  American  (B.  A.  Branum,  personal  commimication,  September  6, 1995). 

Prior  Mathematics  Achievement.  The  percentage  correct  on  an  Algebra/Trigonometry 
placement  exam  was  used  as  the  measure  of  prior  mathematics  achievement.  The  placement 
exam  contained  35  multiple  choice  items  (25  algebra  items  and  10  trigonometry  items)  and 
was  machine  scored.  The  test  was  validated  for  content  in  1995  by  faculty  members  of  the 
USAFA  math  placement  team.  The  tests  were  found  to  have  high  predictive  validity  for 
placing  students  into  Precalculus  as  their  first  mathematics  course,  with  87%  of  students 
successfully  completing  Precalculus  with  a  grade  of  B+  or  less  (A’s  and  A-’s  were  considered 
erroneously  placed;  W.  A.  Kiele,  personal  communication,  April  5, 1995).  Many  of  the 
placement  test  items  are  anchored,  that  is,  used  again  fi-om  year  to  year.  The  use  of  anchored 
items  improves  test  stability  and  reliability. 

The  placement  exams  were  administered  under  standardized  conditions  a  few  days 
after  the  students  arrived  at  the  Air  Force  Academy.  Students  took  the  exam  in  large  lecture 
halls  proctored  by  instructors.  Standardized  directions  were  printed  on  the  first  page  of  the 
exam  and  read  aloud  by  the  proctors.  All  students  had  identical  time  limits.  The  use  of 
calculators  was  not  permitted. 
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Mathematics  Anxiety.  Mathematics  anxiety  was  measured  by  a  subset  of  items  jfrom 
the  Math  Anxiety  Rating  Scale  (MARS),  college  and  adult  version  (Suinn,  1972).  The 
MARS  is  a  98-item  self-rating  scale  set  in  a  five  point  Likert  format  designed  as  a  diagnostic 
or  screening  tool  for  measuring  mathematics  anxiety.  Scores  on  each  MARS  item  represent 
the  level  of  anxiety  reported  for  a  specific  situation.  Selections  range  from  1  representing 
"not  at  all  anxious"  to  5  representing  "very  much  anxious."  An  overall  mathematics  anxiety 
score  is  achieved  by  summing  the  individual  item  scores. 

Since  its  publication  in  1972,  the  MARS  has  been  the  prevailing  instrument  for 
measuring  mathematics  anxiety  (Alexander  &  Martray,  1989).  Alexander  and  Martray  (1989) 
used  a  two-staged  factor  analysis  to  develop  an  abbreviated  version  of  the  MARS.  Their  first 
factor  analysis  reduced  the  98-item  MARS  to  69  items  by  selecting  the  items  most  highly 
correlated  to  each  of  five  identified  factors.  The  69-item  MARS  was  again  abbreviated  by 
application  of  factor  analysis.  Items  that  correlated  highly  with  each  of  three  identified 
factors  were  selected  for  Alexander  and  Martray’s  25-item  abbreviated  MARS.  The  25-item 
MARS  was  shown  to  have  high  internal  consistency  within  each  of  the  three  factors 
(Cronbach  alpha  of  .96,  .86,  and  .84,  respectively).  In  addition,  correlation  between  the  25- 
item  and  69-item  versions  of  the  MARS  was  found  to  be  high  (r  =  .93)  and  test-retest 
reliability  after  two  weeks  was  also  high  (r  =  .86).  Alexander  and  Martray  (1989)  declared 
that  the  25-item  MARS  was  a  “psychometrically  equivalent  alternative”  to  the  98-item 
MARS,  while  being  more  efficient,  less  costly,  and  easier  to  implement  (p.  149). 

The  abbreviated  MARS  was  administered  to  the  control  and  treatment  groups  during 
the  fifth  week  of  class.  A  standardized  set  of  instructions  was  read  aloud  by  the  instructors. 
Students  were  assured  that  their  instructors  would  not  have  access  to  the  individual  MARS 
scores.  The  surveys  were  machine  scored. 

Precalculus  Achievement.  Six  variables  were  used  to  measure  student  achievement  in 
Precalculus.  Included  were  four  hourly  exams,  a  final  exam,  and  the  final  course  percentage 
grade.  The  second,  third,  and  fourth  hourly  exams  included  mostly  new  material  with  a  few 
(20%)  items  testing  material  covered  on  earlier  exams.  The  final  exam  was  comprehensive. 
All  exam  items  were  written  by  members  of  the  USAFA  Department  of  Mathematical 
Sciences  and  the  same  exam  was  administered  to  all  sections.  Parallel  make-up  exams  were 
administered  to  the  few  students  who  missed  an  exam.  All  exams  were  composed  of  multiple 
choice  and  open-ended  items.  The  exams  were  reviewed  by  several  mathematics  instructors 
for  content  validity.  Split-half  reliability  coefficients  for  all  exams  were  calculated  using  the 
Spearman-Brown  prophecy  formula  (Fraenkel  &  Wallen,  1993)  and  were  found  to  be 
acceptable  (coefficients  ranged  from  .69  to  .83). 

As  standard  procedure  at  the  Air  Force  Academy,  exams  were  administered  to  the 
entire  course  population  during  the  same  period  of  time.  Students  were  assigned  to  lecture 
halls  and  classrooms.  Standardized  directions  were  printed  on  the  first  page  of  the  exams  and 
read  aloud  by  the  instructors  administering  the  exam.  All  students  had  identical  time  limits. 

The  four  hourly  exams  were  given  from  7:00  to  7:50  a.m.,  before  the  start  of  classes. 
Students  in  both  the  treatment  and  control  groups  were  permitted  to  use  calculators  on  all  four 
hourly  exams. 

The  final  exam  was  given  seven  days  after  the  last  class  and  was  administered  in  two 
parts.  Students  were  given  1  hour  to  complete  Part  I  of  the  exam  and  2  hours  and  50  minutes 
to  complete  Part  II.  With  the  exception  of  five  items.  Part  I  was  identical  to  the 
Algebra/Trigonometry  Placement  Exam.  Part  II  was  a  cumulative  exam  containing  mostly 
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anchored  items.  Students  were  not  permitted  to  use  calculators  on  Part  I  of  the  final  exam. 
The  use  of  calculators  was  permitted  on  Part  II. 

Multiple  choice  exam  items  for  all  exams  were  machine  scored.'  Standardized  rubrics 
were  used  to  score  open-ended  items.  In  most  cases,  one  instructor  was  assigned  to  score  one 
item  on  all  exam  papers.  For  exam  items  that  were  scored  by  more  than  one  instructor,  a 
sample  of  30  exams  (15  from  the  treatment  group  and  15  from  the  control  group)  was  selected 
for  duplicate  scoring.  Inter-scorer  reliability  was  calculated  and  fovuid  to  be  high  (correlation 
coefficients  ranged  from  .87  to  .99).  All  exam  scores  were  converted  to  percentages. 

The  final  course  percentage  grade  was  based  on  the  folloAving  sub-scores:  (a)  four 
hourly  exams,  45%;  (b)  final  exam,  30%;  (c)  three  written  exercises,  5%;  (d)  course  project, 
5%;  (e)  three  group  problem  solving  exercises,  5%;  and  (f)  quiz,  homework,  and  participation 
points  awarded  by  the  individual  instructors,  10%. 

Procedures.  The  experiment  employed  the  ATI  compensatory  instructional  model. 
The  distributed  practice  treatment  was  designed  to  interact  with  the  lowjprior  achievement 
and  high  mathematics  anxiety  student  aptitudes  by  circumventing  or  neutralizing  their 
debilitating  effects  (Salomon,  1972).  As  recommended  in  previous  ATI  and  homework 
research,  the  duration  of  the  treatment  was  one  semester,  the  entire  duration  of  the  Precalculus 
course  (Austin,  1979;  Becker,  1970;  Becker  &  Yoxmg,  1978;  Cronbach  &  Snow,  1977; 
Holtan,  1982;  Koran  &  Koran,  1984;  Snow,  1977). 

Although  assignment  to  Precalculus  sections  was  not  purely  random,  student  course 
schedules  at  the  USAFA  are  computer  generated  and  students  (especially  first  year  students) 
have  very  few  choices  in  their  schedules.  The  treatment  group  consisted  of  approximately 
46%  of  the  Precalculus  students  (161  students  divided  into  eight  sections).  The  control  group 
consisted  of  the  remaining  students  enrolled  in  Precalculus  (190  students  divided  into  nine 
sections). 

To  minimize  instructor  workload,  each  instructor  was  assigned  either  all  treatment 
sections  or  all  control  sections.  The  Precalculus  sections  were  taught  by  eight  different 
instructors;  three  instructors  taught  treatment  group  sections  and  five  instructors  taught 
control  group  sections. 

All  instructors  were  active  duty  members  of  the  United  States  Air  Force.  Degree 
levels  for  instructors  ranged  from  bachelor  to  doctoral  with  most  instructors  holding  a  master 
of  science  degree.  Instructor  experience  level  varied  from  first  year  instructors  to  a  seasoned 
instructor  with  over  20  years  teaching  experience.  Although  most  of  the  instructors  had  some 
prior  teaching  experience,  few  had  prior  experience  teaching  Precalculus.  Both  experienced 
and  inexperienced  instructors  were  assigned  to  each  group  in  an  attempt  to  equalize  instructor 
experience  across  groups.  When  weighted  by  the  number  of  sections,  the  mean  instructor 
experience  level  for  each  group  was  2.6  years.  The  median  experience  level  was  2  years. 

The  course  topics,  textbook,  handouts,  reading  assignments,  and  graded  assignments 
(with  the  exception  of  quiz,  homework,  and  participation  points)  were  identical  for  the 
treatment  and  control  groups.  The  listing  of  homework  assignments  in  the  syllabus  differed 
between  groups.  The  control  group  was  assigned  daily  homework  related  to  the  topic(s) 
presented  that  day  in  class.  Peterson  (1971)  calls  this  the  vertical  model  for  assigning 
mathematics  homework.  The  treatment  group  was  assigned  homework  in  accordance  with  a 
distributed  organizational  pattern  that  combines  practice  on  current  topics  and  reinforcement 
of  previously  covered  topics.  Under  the  distributed  model,  approximately  40%  of  the 
problems  on  a  given  topic  were  assigned  the  day  the  topic  was  first  introduced,  with  an 
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additional  20%  assigned  on  the  next  lesson  and  the  remaining  40%  of  problems  on  the  topic 
assigned  on  subsequent  lessons  (Hirsch  et  al.,  1983).  In  Hirsch’s  research  and  in  this  study, 
after  the  initial  homework  assignment,  problem(s)  representing  a  given-topic  resurfaced  on  the 
2nd,  4th,  7th,  12th,  and  21st  lesson.  Consequently,  treatment  group  homework  for  lesson  one 
consisted  of  only  one  topic;  homework  for  lessons  two  and  three  consisted  of  two  topics;  and 
homework  for  lesson  four  through  six  consisted  of  three  topics.  This  pattern  continued  as 
new  topics  were  added  and  was  applied  to  all  non-exam,  non-laboratory  lessons. 

The  same  homework  problems  were  assigned  to  both  groups  with  only  the  pattern  of 
assignment  differing.  Because  of  the  nature  of  the  distributed  practice  model,  homework  for 
the  treatment  group  contained  fewer  problems  (relative  to  the  control  group)  early  in  the 
semester  with  the  number  of  problems  increasing  as  the  semester  progressed.  Later  in  the 
semester,  homework  for  the  treatment  group  contained  more  problems  (relative  to  the  control 
group).  By  the  end  of  the  semester,  both  groups  had  been  assigned  precisely  the  same 
homework  problems. 

Because  homework  was  the  key  manipulated  variable  in  this  experiment,  and  because 
larger  effects  on  achievement  were  sometimes  found  when  homework  was  graded  (Austin, 
1979;  Lai,  1994;  Paschal,  Weinstein,  &  Walberg,  1984),  instructors  were  directed  to  collect 
all  homework.  Homework  was  checked  and  coded  for  correctness  and  completion  on  a  three 
point  scale  (0  =  less  than  one-third  complete  and  correct,  1  =  one-third  to  two-thirds  complete 
and  correct,  and  2  =  more  than  two-thirds  complete  and  correct). 

Instructors  in  both  groups  were  encouraged  to  use  class  time  to  discuss  and  review  the 
assigned  homework  problems.  Prior  to  the  second,  third,  and  fourth  exam,  and  at  the  end  of 
the  semester,  both  groups  spent  one  lesson  in  review.  Review  lessons  were  planned  by  the 
individual  instructors.  Classroom  observations  and  student  and  instructor  surveys  were  used 
to  ensure  that  the  treatment  was  administered  as  planned  and  directed. 

Results 

The  means  and  standard  deviations  for  the  entire  sample  and  for  the  treatment  and 
control  groups  on  measures  of  prior  achievement,  mathematics  anxiety,  and  Precalculus 
achievement  are  reported  in  Table  1. 

Hierarchical  multiple  regression  was  employed  to  test  the  hypotheses^.  Three  sets  of 
independent  variables  were  defined.  Set  A,  the  covariate  set,  contained  two  variables:  (a) 
prior  math  achievement,  and  (b)  mathematics  anxiety.  Set  B  contained  the  group  membership 
variable  (treatment  group  or  control  group).  Set  C,  the  two-way  interaction  set,  contained  two 
interaction  variables:  (a)  Prior  Achievement  x  Treatment,  and  (b)  Anxiety  x  Treatment.  The 
dependent  variable  in  this  study  was  Precalculus  achievement.  Precalculus  achievement  was 
measured  as  the  semester  progressed  and  produced  six  scores:  four  hourly  exam  scores,  a  final 
exam  score,  and  a  final  course  percentage  grade.  By  analyzing  each  measure  of  achievement 
separately,  the  goal  was  to  determine  whether  the  length  of  treatment  had  an  impact  on 
achievement  with  the  expectation  that  the  distributed  practice  treatment  would  have  a 
cumulative  effect  (Austin,  1979). 


^  Multiple  regression  was  selected  as  the  analysis  tool  due  to  its  ability  to  handle  unequal  cell  sizes  and 
quantitative  independent  variables  (Cohen  &  Cohen,  1983). 
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Table  1.  Descriptive  statistics  for  measures  of  prior  achievement,  anxiety,  and  precalculus 
achievement.  All  prior  achievement  and  achievement  scores  are  measured  in  percent. 


Prior 

Achievement 

Math 

anxiety 

isl 

Exam 

Exam 

3™ 

Exam 

— 

Exam 

Final 

Exam 

Course 

Grade 

N 

351 

351 

All 

351 

Students 

341 

338 

333 

317 

333 

Mean 

35.88 

51.51 

80.43 

70.67 

70.48 

65.21 

70.43 

74.83 

SD 

8.74 

14.44 

13.25 

13.67 

13.10 

13.55 

11.13 

8.55 

Min 

5.00 

28.00 

14.81 

21.48 

29.63 

23.70 

20.33 

35.00 

Max 

50.00 

99.00 

99.26 

96.30 

100.00 

100.00 

94.67 

96.76 

N 

161 

161 

Treatment 

161 

Group 

160 

157 

155 

144 

155 

Mean 

36.51 

49.48 

82.69 

73.58 

70.71 

68.2-8 

71.70 

76.96 

SD 

8.09 

12.96 

11.89 

12.79 

12.99 

12.73 

10.60 

7.84 

Min 

5.00 

28.00 

28.99 

37.78 

29.63 

23.70 

28.61 

46.43 

Max 

50.00 

93.00 

99.26 

95.56 

98.52 

100.00 

93.56 

94.83 

N 

190 

190 

Control 

190 

Group 

181 

181 

178 

173 

178 

Mean 

35.36 

53.23 

78.51 

68.10 

70.27 

62.54 

69.41 

72.97 

SD 

9.24 

15.42 

14.05 

13.93 

13.23 

13.71 

11.48 

8.72 

min 

5.00 

28.00 

14.81 

21.48 

30.37 

28.15 

20.33 

35.00 

max 

47.50 

99.00 

99.26 

96.30 

100.00 

99.26 

94.67 

96.76 

Hypothesis  Testing.  Table  2  shows  the  results  of  the  step-by-step  hierarchical 
regressions  as  the  three  sets  of  independent  variables  were  added. 

Effect  of  the  Covariates.  Step  one  of  the  hierarchical  mxiltiple  regression  analyses 
tested  the  effect  of  the  covariates  (Set  A,  prior  mathematics  achievement  and  mathematics 
anxiety)  on  Precalculus  achievement.  Set  A  was  regressed  on  each  of  the  six  measures  of 
Precalculus  achievement.  A  significant  proportion  of  variance  in  all  six  measures  of 
Precalculus  achievement  was  explained  by  prior  mathematics  achievement  and  mathematics 
anxiety  (see  Table  2). 

Main  Treatment  Effect.  Step  two  of  the  hierarchical  analyses  tested  for  a  main  effect 
due  to  the  distributed  practice  treatment.  The  covariates  (Set  A)  and  the  group  membership 
variable  (Set  B)  were  regressed  on  each  of  the  six  measures  of  Precalculus  achievement. 

Tests  of  the  semi-partial  correlation  coefficients  revealed  that,  when  the  co  variates  were 
controlled  for,  the  distributed  practice  treatment  accounted  for  a  statistically  significant 
proportion  of  the  variance  in  Precalculus  achievement  in  all  but  the  third  exam  and  final  exam 
(see  Table  2). 

Two-Way  ATI  Effects.  Step  three  of  the  hierarchical  regression  analysis  added  the  two 
aptitude-treatment  interaction  variables  (Set  C).  The  semi-partial  correlation  coefficients 
were  tested  to  determine  whether  the  interactions  accounted  for  any  variance  in  Precalculus 
achievement  above  what  had  already  been  accotmted  for  by  prior  achievement,  anxiety,  and 
the  distributed  practice  treatment.  The  effect  of  the  two-way  ATIs  was  not  statistically 
significant  for  any  of  the  six  measures  of  Precalculus  achievement  (see  Table  2). 
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Table!.  Hierarchical  multiple  regression  analysis:  main  effect  and  interaction  effect.  Set  A 
=  placement  test  score  and  math  anxiety  score.  Set  B  =  group  membership.  Set  C  =  two-way 
interactions.  *e<.05  **2<.01  ***p<.001 


Independent 
variable  sets 

Cumulative 

df 

F 

Variable 

sets 

added 

Increment 

toR! 

F  of  the 
increment 

First 

Exam 

A 

.239 

2,  348 

54.66*** 

A 

A,B 

.249 

B 

.010 

1,347 

4.73* 

A,B,C 

.251 

C 

.001 

2, 345 

0.30 

Second 

Exam 

A 

.169 

2,338 

34.42*** 

A 

A,B 

.193 

B 

.023  -  - 

1,337 

9.78** 

A,B,C 

.198 

C 

.005 

2,335 

1.07 

Third 

Exam 

A 

.069 

2, 335 

12.46*** 

A 

A,B 

.070 

B 

.000 

1,334 

0.13 

A,  B,C 

.073 

C 

.003 

2,332 

0.56 

Fourth 

Exam 

A 

.093 

2,330 

16.97*** 

A 

A,B 

.124 

B 

.031 

1,329 

11.57** 

A,B,C 

.128 

C 

.004 

2, 327 

0.71 

Final 

Exam 

A 

.121 

2,314 

21.61*** 

A 

A,B 

.125 

B 

.004 

1,313 

1.27 

A,B,C 

.126 

C 

.001 

2,311 

0.24 

Final 

Grade 

A 

.203 

2,330 

41  90*** 

A 

A,B 

.234 

B 

.031 

1,329 

13.48** 

A,B,C 

.236 

C 

.002 

2, 327 

0.36 

Instructor  Effects.  Regression  analysis  was  also  used  to  determine  whether  there  was 
a  significant  effect  due  to  instructor  after  prior  achievement,  anxiety,  and  the  treatment  were 
controlled  for.  A  two-step  hierarchical  regression  was  employed  with  the  covariate  and  group 
membership  variables  (Set  A’)  entered  in  the  first  step  and  the  dummy-coded  instructor 
variable  set  (Set  B’)  added  in  the  second  step.  Semi-partial  correlation  coefficients  were 
calculated  and  F-tests  were  conducted.  This  analysis  revealed  that  the  instructors  did  not 
contribute  to  the  variance  in  Precalculus  achievement  beyond  what  was  already  accounted  for 
by  prior  achievement,  anxiety,  and  the  distributed  practice  treatment. 

Study  Time.  The  USAFA  routinely  collects  study  time  data.  After  each  exam,  a  large 
sample  of  cadets  (at  least  60%  of  the  course  population)  anonymously  reported  the  amount  of 
time  (in  minutes)  spent  studying  for  the  exam.  Time  spent  studying  was  approximately  equal 
for  both  groups  (see  Table  3).  Descriptive  data  revels  that,  for  both  the  treatment  and  control 
group,  study  time  for  the  third  exam  was  at  least  16%  greater  than  study  time  for  any  other 
exam.  Study  time  for  the  final  exam  was  at  least  68%  greater  than  study  time  for  any  of  the 
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hourly  exams  (see  Table  3).® 

Table  3.  Analysis  of  study  times  for  exams.  All  e  values  >  .20. 


Exam 

Treatment  mean 

Control  mean 

df 

t 

1st  Exam 

88.4  min. 

84.5  min. 

333 

0.59 

2nd  Exam 

95.4  min. 

97.4  min. 

296 

0.23 

3rd  Exam 

117.6  min. 

116.9  min. 

305 

0.08 

4th  Exam 

100.8  min. 

93.2  min. 

274 

0.77 

Final  Exam 

198.1  min. 

235.9  min. 

128 

1.30 

Effect  of  Homework  on  Exam  Scores.  Five  separate  regressions  v^ere  performed  to 
determine  whether  homework  scores  could  predict  a  significant  proportion  of  variance  in 
exam  scores.  Block  homework  scores  explained  a  statistically  significant  proportion  of 
variance  in  all  hourly  exam  scores.  Similarly,  the  total  homework  score  explained  a 
statistically  significant  proportion  of  variance  in  the  final  exam  score  (see  Table  4). 

Table  4.  Effect  of  homework  on  exam  scores.  ***£  <  .001 


Exam 

r 

df 

F 

1st  Exam 

.39 

.151 

1,349 

62.07*** 

2nd  Exam 

.33 

.109 

1,339 

41  54*** 

3rd  Exam 

.33 

.109 

1,336 

41.22*** 

4th  Exam 

.30 

.090 

1,331 

32.67*** 

Final  Exam 

.39 

.153 

1,315 

56.96*** 

Discussion  and  Conclusions 

Distributed  Practice  Effect.  The  distributed  practice  treatment  produced  a  statistically 
significant  main  effect  on  four  out  of  six  measures  of  Precalculus  achievement  (three  hourly 
exams  and  the  final  course  percentage  grade).  These  findings  are  in  agreement  with  results 
reported  by  Friesen  (1975),  Parker  (1990),  Peterson  (1970),  Reed  (1983;  Klingele  &  Reed, 
1984),  and  Saxon  (1982).  The  treatment  did  not  produce  a  statistically  significant  main  effect 
on  the  third  exam  or  final  exam. 

Effect  sizes  were  calculated  to  better  interpret  the  practical  significance  of  the 
distributed  practice  treatment.  The  treatment  produced  an  effect  size  (f^)  of  0.013  on  the  first 
exam,  0.029  on  the  second  exam,  0.035  on  the  fourth  exam,  and  0.040  on  the  final  course 
percentage  grade.  Although  the  effect  sizes  appear  to  be  small,  the  treatment  group  outscored 
the  control  group  in  every  case.  A  mean  difference  of  5.13  percentage  points  on  fte  first, 
second,  and  fourth  exam  translates  to  an  advantage  of  about  a  third  of  a  letter  grade  for 
students  in  the  treatment  group.  In  addition,  higher  minimum  scores  earned  by  the  treatment 
group  may  indicate  that  the  distributed  practice  treatment  served  to  eliminate  the  extremely 
low  scores  (refer  to  Table  3).  As  postulated  by  Austin  (1979),  the  distributive  practice 

®  Since  the  group  of  students  sampled  for  study  time  for  one  exam  was  not  necessarily  independent  of  the  group 
of  students  sampled  for  study  time  for  other  exams,  inferential  statistical  tests  of  study  times  between  exams  are 
not  appropriate. 
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treatment  appeared  to  have  a  cumulative  effect. 

Because  the  distributed  practice  treatment  produced  a  significant  main  effect  on  all  but 
one  of  the  hourly  exams,  a  plausible  explanation  for  this  aberration  was- sought.  The 
treatment  and  control  groups  achieved  nearly  equal  scores  on  the  third  exam  (treatment 
mean  =  70.71  and  control  mean  =  70.27).  Although  the  two  groups  spent  nearly  equal  time 
studying  for  the  exam  (treatment  mean  =  1 17.6  minutes  and  control  mean  =  1 16.9  minutes), 
both  groups  reported  spending  much  more  time  studying  for  the  third  exam  than  they  spent 
studying  for  any  of  the  other  three  hourly  exams.  The  third  exam  occurred  after  mid-semester 
progress  reports  which  may  have  motivated  students  to  devote  more  time  to  studying.  It  is 
possible  that  the  additional  study  time  imitated  the  distributed  practice  treatment  by  allowing 
for  more  repetitions  of  problem  types. 

Oddly,  the  distributed  practice  treatment  did  not  produce  a  significant  effect  on  final 
exam  scores.  One  possible  cause  for  the  disparity  was  the  USAFA  policy  exempting  the  top 
performers  from  the  final  exam.  Of  the  16  exempted  students,  1 1  were  from  the  treatment 
group  with  only  5  from  the  control  group.  It  is  likely  that  the  treatment  group  would  have 
outscored  the  control  group  on  the  final  exam  if  these  top  performers  had  taken  the  exam.  In 
addition,  increased  study  time  for  the  final  exam  may  have  influenced  the  results.  Because 
the  final  exam  was  scheduled  late  during  final  exam  week,  study  time  for  the  exam  was  not 
only  longer,  but  more  widely  distributed.  The  benefits  of  the  longer  and  more  dispersed  study 
time  may  have  been  similar  to  the  benefits  created  by  the  distributed  practice  treatment. 

Aptitude-Treatment  Interaction  Effects.  Two  significant  two-way  interactions  were 
expected:  (a)  Prior  Mathematics  Achievement  x  Treatment,  and  (b)  Mathematics  Anxiety  x 
Treatment.  Neither  of  these  interactions  was  found  to  explain  a  significant  proportion  of 
variance  in  Precalculus  achievement  beyond  what  had  already  been  explained  by  the 
covariates  and  the  distributed  practice  treatment. 

The  sample  in  this  study,  first  year  students  on  the  low  mathematics  ability  track  at  the 
Air  Force  Academy,  may  provide  some  explanation  for  the  lack  of  significant  interaction 
effects.  Students  on  the  average  track  are  typically  enrolled  in  Calculus  I  during  the  Fall 
semester  and  Calculus  II  during  the  Spring  semester.  Similarly,  those  with  high  math  ability 
are  usually  enrolled  in  Calculus  II  or  Calculus  III  during  the  Fall  semester.  Because 
mathematics  achievement  has  been  found  to  correlate  negatively  with  mathematics  anxiety 
(Berenson,  Carter,  &  Norwood,  1992;  Clute,  1984;  Coleman,  1991;  Cooper  &  Robinson, 

1989;  Covington  &  Omelich,  1987;  Frary  &  Ling,  1983;  Gliner,  1987;  Hembree,  1990; 
Lawson,  1993;  McCoy,  1992;  Richardson  &  Suinn,  1972),  the  students  placed  into 
Precalculus  were  probably  relatively  high  in  mathematics  anxiety.  Aptitude-treatment 
interactions  are  not  expected  to  be  as  strong  when  students  have  comparable  aptitudes.  The 
homogeneity  of  this  group  may  have  nullified  the  expected  two-way  interaction  effects. 

The  results  of  this  study  challenge  the  results  reported  by  Hirsch  and  his  colleagues 
(1982, 1983).  Hirsch  et  al.  found  significant  Prior  Achievement  x  Treatment  ATIs  on  three 
out  of  five  measures  of  Calculus  I  achievement.  In  all  three  cases,  the  distributed  practice 
treatment  was  beneficial  to  students  scoring  at  or  below  the  mean  on  an  algebra  and  analytic 
geometry  pre-test.  It  is  not  known  whether  the  students  in  Hirsch’ s  study  were  grouped 
homogeneously. 

Limitations.  This  study  was  limited  by  the  length  of  the  semester  and  the  number  of 
homework  assignments.  By  following  the  homework  pattern  advocated  by  Hirsch  et  al. 

(1982,  1983),  homework  for  topics  introduced  after  the  tenth  lesson  could  not  be  fully 
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distributed.  Homework  for  each  topic  was  assigned  in  the  order  listed  in  the  textbook,  in 
which  the  easier  problems  preceded  the  more  difficult  ones.  For  the  treatment  group,  this 
meant  that  the  easiest  problems  were  assigned  early  in  the  distribution  pattern  with  the  hardest 
problems  assigned  in  the  later  stages  of  the  distribution.  The  treatment  may  have  been  more 
effective  if  the  difficulty  level  of  problems  within  each  assignment  was  mixed.  Similarly,  the 
distributed  practice  treatment  may  be  more  effective  when  applied  to  courses  of  longer 
duration. 

Several  factors  may  limit  the  generalizability  of  this  study.  Although  the  sample  was 
large,  the  subjects,  being  military  academy  cadets,  may  not  be  representative  of  typical  high 
school  or  college  students.  Overall,  students  attending  the  USAFA  are  a  fairly  homogeneous 
group  with  similar  academic  and  career  goals.  The  limited  external  validity  due  to  the 
controlled  atmosphere  at  the  Air  Force  Academy  serves  to  strengthen  the  internal  validity  of 
the  study.  Threats  due  to  subject  characteristics,  mortality,  location,  history,  and  subject 
attitude  have  been  minimized  due  to  the  controlled  environment  at  the  USAFA  (Fraenkel  & 
Wallen,  1993). 

Certain  threats  to  internal  validity  remain.  Although  it  cannot  be  assumed  that 
instructors  with  similar  experience  levels  are  equally  effective,  this  study  and  a  previous  study 
conducted  at  the  USAFA  foimd  that  instructor  experience  was  not  a  significant  contributor  to 
achievement  variance  (Thompson,  Mitchell,  Coffin,  &  Hassett,  1979).  It  is  possible  that  one 
or  more  instructors  were  biased,  either  for  or  against  the  distributed  practice  treatment.  A 
Hawthorne  effect  may  have  resulted  if  the  students  in  the  treatment  group  recognized  that 
they  were  receiving  special  treatment  in  the  way  of  distributed  practice  homework 
assignments  (Fraenkel  &  Wallen,  1993).  Conversely,  students  assigned  to  the  control  group 
may  have  suffered  a  demoralization  effect  (Fraenkel  &  Wallen).  In  addition,  the  treatment 
may  have  had  a  negative  impact  on  the  achievement  of  the  treatment  group  if  exam  items 
were  related  to  homework  problems  not  yet  assigned  due  to  the  distributed  practice  syllabus. 
Finally,  it  is  possible  that  the  treatment  was  not  fully  confined  to  the  treatment  group. 
Although  survey  responses  indicated  that  students  rarely  studied  with  students  who  used  a 
different  syllabus,  it  is  possible  that  cadets  discussed  homework  problems  with  students  from 
other  sections. 

Recommendations  for  Future  Research.  Distributed  practice  homework  has  been 
shown  to  be  beneficial  to  students  on  the  low  mathematics  track  at  the  USAFA.  Testing  of 
the  distributed  practice  treatment  on  medium  and  high  ability  students  is  recommended.  In 
addition,  different  variations  of  spaced  review  should  be  investigated  across  a  wide  variety  of 
students,  institutions,  and  mathematics  courses.  Because  the  collection  and  grading  of 
homework  may  have  caused  a  higher  than  average  homework  completion  rate,  this  study 
should  be  replicated  in  an  environment  where  homework  is  not  collected. 

Future  studies  of  this  kind  should  include  the  study  time  variable.  The  study  time  data 
in  this  experiment  indicate  that  the  distributed  practice  treatment  had  the  greatest  impact  when 
less  time  was  devoted  to  studying  for  an  exam.  This  finding  appears  to  support  the  Aeory  that 
distributed  practice  assignments  receive  more  attention  than  massed  assignments.  An 
analysis  of  how  students  use  their  study  time  could  help  shed  light  on  why  and  how  this 
phenomenon  occurs. 

According  to  Holtan  (1982),  the  value  of  the  distributed  practice  treatment  may  well 
be  in  the  delayed  retention  of  the  skills  and  concepts  practiced.  Follow-up  retention  tests  are 
recommended  for  the  students  taking  part  in  this  study. 
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The  multiple  correlations  revealed  in  this  study  accoimted  for  less  than  26%  of  the 
variance  in  all  measures  of  achievement.  This  suggests  that  the  contribution  of  other  variables 
such  as  motivation,  attitude,  and  study  habits  should  be  examined.  Systematic  research  in  this 
area  should  help  identify  the  students  who  will  benefit  most  from  distributed  practice 
assignments  and  contribute  to  the  theoretical  structure  of  ATI. 

Summary 

This  study  documented  a  significant  positive  correlation  between  homework  scores 
and  exam  scores.  Homework  scores  were  found  to  account  for  between  10%  and  15%  of  the 
variability  in  exam  scores.  Meaningful  homework  may  be  viewed  as  an  important  component 
in  mastering  mathematics  course  material. 

Enrollments  in  remedial  mathematics  college  courses  are  on  the  rise  (Berenson  et  al., 
1992)  and  90%  of  college  mathematics  enrollments  are  in  elementary  calculus,  elementary 
statistics,  and  courses  prerequisite  to  them  (National  Research  Council,  1989).  There  is  great 
potential  for  application  of  die  distributed  practice  model.  Mathematics  achievement  is  still 
the  principal  gateway  for  students  preparing  to  enter  technical  and  scientific  careers,  and 
distributed  practice  may.help  foster  success  in  these  pivotal  math  courses. 
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In-Class  Exercises  for  Active  Learning  and  Beyond 

Kenneth  J.  Soda,  Ph.D. 

Mark  C.  Crews,  Ph.D.,  Lt  Colonel,  USAF 

Department  of  Electrical  Engineering 

Abstract:  The  In-Class  Exercise  has  gained  wide  acceptance  in  the  US  Air  Force  Academy 
Department  of  Electrical  Engineering  as  a  tool  through  which  active  learning  can  be  achieved  in  the 
classroom.  After  some  years  of  use,  it  has  become  clear  that  the  utility  of  these  exercises  extends  well 
beyond  their  original  intent.  Important  windfall  benefits  have  been  realized  in  course  development, 
evaluation,  instructional  efficiency  and  development  of  new  instructors.  We  feel  their  advantages 
make  them  worthy  of  careful  consideration  by  teaching  professionals  of  all  disciplines. 

No  one  need  remind  a  USAF  Academy  (USAF  A)  faculty  member  of  the  diversity  of 
activities  in  which  their  students  engage  themselves.  In  a  typical  semester,  instructors  have 
face-to-face  contact  with  their  cadet  students  less  than  2%  of  their  waking  hours’.  It  is 
therefore  exceedingly  important  that  we  maximize  the  effectiveness  of  those  precious  minutes 
of  classroom  instruction.  We,  in  the  Department  of  Electrical  Engineering  (DFEE),  have 
found  the  In-Class  Exercise  (ICEx)  to  be  an  important  tool  for  improving  instructional 
effectiveness.  Over  one-third  of  our  active  classroom  courses  (versus  laboratory  or  design 
courses)  use  these  exercises,  including  DFEE’s  core  course*,  El  Engr  215.  Originally 
intended  as  a  method  to  invoke  active  learning  processes,  these  exercises  have  evolved  into 
much  more.  Through  years  of  use  it  has  become  clear  how  these  exercises  also  affect  our 
organization  of  course  materials,  our  choice  of  course  content,  the  evaluation  our  students, 
and  the  development  of  new  instructors.  We  have  come  to  recognize  their  value,  not  only  as 
aids  to  teaching,  but  to  teacher  effectiveness  as  well. 

The  In-Class  Exercise:  What  and  Why 

Although  the  exact  nature  of  these  exercises  varies  from  course  to  course,  our  In-Class 
Exercises  have  evolved  into  a  fairly  consistent  form.  Naturally,  our  ICExs  are  delivered  in  an 
active  learning  classroom  environment.  The  content  of  each  exercise  is  chosen  to  highlight 
key  concepts  or  principles  under  consideration.  These  exercises  are  generally  adapted  from 
problems  or  examples  found  in  the  course  text  or  materials.  They  are  limited  in  scope  so  they 
may  be  completed  within  a  single  class  period,  ideally  in  less  than  twenty  minutes.  Most  are 
delivered  on  a  single  page.  All  exercises  are  accompanied  by  printed  solutions  released  after 
completion  of  the  exercise.  Each  exercise  requires  a  written  response  from  every  student.  An 
example  of  one  such  In-Class  Exercise  is  found  in  Appendix  A. 

There  are  a  few  basic  reasons  for  the  form  into  which  ICExs  have  evolved.  Their 
fundamental  purpose  is  to  draw  cadets  into  an  active  role  in  the  classroom  and  to  engage  them 
with  key  concepts  or  processes.  Their  content  is  selected  so  that  a  strong  pedagogical  tie  is 
made  with  other  course  resources.  In  this  way  ICExs  serve  as  introductions  or  reinforcements 
to  the  more  detailed  discussion  provided  by  the  text  or  course  materials.  We  also  believe  the 


’  This  estimate  assumes  she  eighteen-hour  days  per  week  and  one  50-minute  section  two  and  one-half  times 
during  that  week.  This  is  the  normal  USAFA  instructional  cycle. 

*  The  USAFA  curriculum  contains  some  112  semester  hours  of  essential  or  “core”  courses  required  of  all  cadets 
for  graduation.  These  courses  have  enrollments  of  some  1,000  cadets  each  academic  year. 
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exercises  must  be  brief  enough  to  capture  and  hold  the  attention  of  our  cadets.  The  single  page 
format  is  helpful  in  bounding  the  complexity  of  the  exercises  and  preventing  them  from 
appearing  too  overwhelming.  We  also  favor  the  distribution  of  printed  solutions  to  our  ICExs 
as  a  method  to  reinforce  concepts  and  reassure  students.  The  solutions  also  offer  instructors  a 
method  to  highlight  issues  not  central  to  the  focus  of  the  exercise,  but  which  are  easily 
illustrated  in  the  framework  of  the  exercise. 

We  have  come  to  favor  the  written  format  for  ICExs  for  five  reasons.  First,  it  creates  a 
permanent  record  of  each  student’s  work  that  may  be  both  evaluated  by  the  instructor  and 
reviewed  by  the  student.  Second,  written  responses  eliminate  the  potential  for  the 
embarrassment  inherent  in  exercises  worked  at  the  blackboard.  Third,  the  graphics  delivered 
with  the  ICExs  streamline  classroom  interactions.  There  is  no  need  to  re-copy  equations, 
figures  or  graphs  from  a  blackboard  or  transparency.  Cadets  are  free  to  focus  instead  upon 
the  problem  or  concept  at  hand.  Fourth,  the  written  exercises  afford  those  cadets  who  might 
miss  a  class  meeting  a  snapshot  of  that  lesson’s  key  concepts.  Finally,  the  written  ICExs  form 
a  permanent  conceptual  record  that  can  be  referred  to  later  in  a  course,  or  in  a  follow-on 
course. 

Although  it  does  not  occur  universally,  we  have  observed  an  additional,  unintended 
benefit  of  written  ICExs.  In  many  cases,  our  cadets  extend  the  written  responses  required  for 
the  ICExs  into  class  notes  (See  Appendix  A).  We  see  many  exercises  where  cadets  will  record 
related  concepts  from  the  lecture,  which  are  not  parts  of  the  exercise.  They,  in  effect,  use  the 
exercises  as  note  taking  aids.  Considering  the  framework  the  exercises  provide,  these  notes 
are  often  more  accurate  and  meaningful  to  students  than  those  they  generate  individually. 

A  few  variations  in  the  delivery  of  these  exercises  are  common  in  our  department. 
Some  courses  award  grade  credit  for  these  exercises,  others  do  not.  Some  exercises  are  run 
with  full  student-to-student  collaboration  where  the  instructor’s  discussion  of  the  exercise  is 
limited.  Others  are  worked  interactively  with  students  selected  by  the  instructor,  with  the 
balance  of  the  class  as  witnesses. 

The  In-Class  Exercise:  Are  They  Beneficial? 

Both  survey  data  and  anecdotal  evidence  indicate  the  positive  perceived  benefit  of 
these  exercises.  A  recently  completed  study  of  teaching  methodologies  reveals  a  strong  cadet 
preference  for  ICExs  (Soda,  1999).  All  cadets  surveyed  in  El  Engr  341  last  semester  rated  the 
In-Class  Exercise  as  the  most  beneficial  of  the  teaching  methods  used  in  this  course.  This 
ICExs  were  preferred  over  lecture,  homework,  and  personal  study  from  the  course  text  or 
outside  references.  While  the  motivations  for  this  preference  varied,  course  critiques  reveal  a 
strong  perceived  utility  as  guides  for  test  preparation.  The  opportunity  to  work  an  exercise 
with  some  measure  of  instructor  guidance  is  another  common  motivation.  Instant  feedback 
relative  to  common  misconceptions  in  a  relatively  low  threat  framework  is  yet  another. 

One  might  well  wonder  if  this  perceived  benefit  is  unique  to  the  USAFA  teaching 
environment.  The  aforementioned  study  mcludes  survey  data  taken  from  students  in  a  similar 
course  taught  at  Colorado  Technical  University  (CTU).  While  not  nearly  as  uniform  in  their 
responses,  the  primarily  part-time  students  at  CTU  felt  ICExs  were  at  least  as  beneficial  to 
learning  as  lecture,  homework  and  self-study.  We  believe  ICExs  hold  inherent  benefit  even  in 
teaching  environments  comprised  primarily  of  part-time  students. 
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In-Class  Exercises:  What  Other  Benefits? 

While  the  ICEx  is  a  powerful  classroom  teaching  tool,  we  have  also  come  to 
appreciate  their  usefulness  from  our  perspective  as  faculty  members.  After  some  years  of  use, 
we  have  come  to  realize  their  merit  as  guides  in  the  selection  of  course  content;  as  a  basis  for 
setting  realistic  limits  on  classroom  discussion;  as  natural  transitions  to  other  course  exercises; 
as  methods  to  express  alternative  explanations;  as  conduits  of  course  materials  to  new 
instructors;  and  as  guides  in  our  evaluation  processes. 

Our  challenge,  indeed  that  of  all  teachers,  is  to  identify,  communicate,  illustrate, 
amplify  and  instill  the  most  important  concepts  and  relationships  of  the  material  at  hand. 

None  of  us  has  the  time  to  teach  all  the  concepts  we  would  like,  hence,  identifying  the  most 
important  ones  is  a  key  process.  Since  ICExs,  by  their  nature,  focus  upon  these  very  concepts, 
we  have  found  their  development  a  convenient  process  through  which  we  may  solidify  our 
topical  choices  for  classroom  presentation.  They  also  serve  as  a  global  summary  of  our 
choices.  A  collection  of  ten  or  twelve  ICExs  laid  side-by-side,  can  more  succinctly  illustrate 
the  flow  of  a  block  of  instruction  than  can  an  equal  number  of  lesson  plans. 

As  long  as  we  are  true  to  the  principle  of  exercise  completion  during  a  single  class 
meeting,  we  are  also  forced  to  carefully  limit  the  content  of  each  lesson.  As  we  develop  new 
exercises,  we  must  constantly  come  to  grips  with  the  teacher’s  dilemma:  what  will  and  won’t 
we  have  the  time  resource  to  cover. 

When  properly  chosen,  ICExs  can  act  as  springboards  to  the  more  intricate  and 
interrelated  concepts  illustrated  in  homework,  computer  simulations  and  laboratory  exercises. 
By  clarifying  the  basics,  our  instructors  use  them  to  build  links  to  other  kinds  of  out-of-class 
work.  Consider  the  computer  based  circuit  simulation  projects  which  are  common  in  our 
department.  ICExs  can  serve  as  illustrations  of  the  operation  of  a  basic  circuit,  which  is  then 
expanded  out  of  class  into  a  more  complex  circuit  in  the  computer  simulation.  We  find  this 
kind  of  transition  useful  in  clarifying  circuit  behavior  and  setting  proper  expectations  for  the 
outcome  of  the  computer  exercise. 

Experienced  teachers  often  develop  their  own  alternative  explanations  for  troubling 
concepts.  In-Class  Exercises  can  act  as  a  convenient  technique  for  recording  and  expressing 
these  alternative  explanations  in  a  fashion  which  is  familiar  to  students.  In  a  sense,  tihese 
exercises  become  the  method  through  which  teachers  can  modify  the  thought  process  laid 
down  by  the  course  text  where  the  content  or  examples  have  fallen  below  expectations. 

There  are  probably  few  post  secondary  educational  institutions  that  have  a  more 
dynamic  faculty  staffing  environment  than  does  USAFA.  Perhaps  as  many  as  one  third  of  the 
military  faculty  members  are  transferred  back  to  operational  assignments  each  academic  year. 
With  this  situation  in  mind,  we  have  found  the  ICEx  as  a  conduit  by  which  detailed  course 
materials  can  be  passed  from  experienced  to  new  instructors.  While  all  teachers  improve  their 
instructional  quality  over  time,  our  new  faculty  find  ICExs  particularly  useful  when  preparing 
to  teach  a  new  course.  Since  they  serve  as  models  of  instructional  techniques,  these  ICExs 
also  serve  as  guides  to  new  faculty  working  to  develop  their  own  course  materials. 

An  often-overlooked  merit  of  the  In-Class  Exercises  is  their  contribution  to  the 
evaluation  process.  When  choosing  topics  for  test  purposes,  the  current  ICExs  can  again  serve 
as  an  important  organizational  tool.  \^^le  our  test  questions  are  virtually  never  identical  to 
ICExs,  we  find  them  a  convenient  summary  of  the  current  field  of  topics  and  principles.  In 
this  framework,  we  can  easily  identify  those  problems  which  clearly  fall  outside  of  the  scope 
of  the  current  block  of  instruction.  Aside  from  the  topical  selection  process,  ICExs  also  serve 
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as  a  reminder  of  the  solution  methods  and  processes  most  heavily  emphasized.  We  have  found 
our  selection  of  problems,  and  their  level  of  difficulty,  to  be  greatly  enhanced  by  a  review  of 
these  instructional  tools. 

In-Class  Exercises:  At  What  Cost? 

No  instructional  process  comes  without  cost.  ICExs  are  not  free  by  any  means. 
Development  of  these  exercises  requires  preparation  extending  beyond  conventional  lesson 
plarming.  Layout,  proofireading,  development  of  solutions  and  reproduction  time  must  all  be 
taken  into  account.  In  those  courses  where  ICExs  are  awarded  course  credit,  grading  is  added 
to  the  burden  of  instruction.  The  effort  these  exercises  demanded  of  instructors  must  be 
repeated  each  time  the  companion  text  or  software  is  modified  or  updated.  The  level  of  effort 
required  to  update  can  affect,  perhaps  inappropriately,  decisions  to  adopt  new  course 
materials. 

Summary 

In-Class  Exercises  offer  a  host  of  advantages  to  our  students:  motivation  to  be  on  task, 
thoughtful  topical  summaries,  and  instant  feedback.  At  the  same  time,  these  exercises  also 
hold  significant  advantages  for  faculty;  guidance  in  the  selection  of  course  content;  a  basis  for 
limiting  classroom  discussion;  springboards  to  other  course  work;  a  conduit  for  alternative 
explanations;  a  legacy  for  new  faculty  to  consider;  and  guidance  for  course  and  test  planning. 
In  a  world  where  the  quantity  of  knowledge  doubles  every  few  years,  educators  must  carefully 
choose  not  only  what  to  teach,  but  how  to  teach.  While  they  demand  effort  to  develop,  present 
and  evaluate,  we  feel  In-Class  Exercises  merit  the  thoughtful  consideration  of  teaching 
professionals  of  all  disciplines. 

Reference 
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Students.  Proceedings  of  the  American  Society  of  Engineering  Educators  1999  Annual 
Conference.  Session  2352. 
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Appendix  A  -  Example  ICEx 


An  example  of  a  completed  In-Class  Exercise  is  shown  below.  This  particular  exercise 
is  taken  from  the  first  block  of  instruction  of  El  Engr  342,  Electronics  II  and  is  used  to  help 
introduce  the  concept  of  differential  pair  amplifiers.  The  cadet  completing  this  particular 
exercise  added  notes  derived  from  the  classroom  discussion  occurring  botih  before  and  after 
the  exercise  was  administered.  These  notes  fall  outside  the  normal  scope  of  this  exercise  and 
represent  an  unintended  of  their  use  (see  text). 


Nam»: 


El  Engr  342  -  In  Cla&s  Exercise: 

Oollaboratl  on  Policy:  You  may  work  with  assigned  cadets  in  the  completion  of 

this' exercise. 

Documentation:  ‘ 

Lesson  2:  Part  A.  Consider  the  diPerential  ampiricer  shown  .below.  .  If  ].  —  1  mA, 
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INFORMATION  TECHNOLOGY 


Economics  Electronic  Workbook  and  Online  Experiments  (Workshop) 

R.  David  Mullin,  Ph.D. 

Department  of  Economics  and  Geography 

Abstract:  This  project  develops  and  disseminates  instructional  software  architecture  that  will 
integrate  empirical  investigation  and  the  experimental  method  into  the  undergraduate  economics 
curriculum.  Users  of  the  software,  which  will  be  available  over  the  internet,  will  be  able  to  add  a 
regular  lab  component  to  their  introductory  and  intermediate-level  courses  in  economics.  The 
laboratory  modules  will  be  powered  by  Microsoft  BackOffice  and  Imprise  Delphi,  and  allow  faculty  to 
easily  add  customizing  features,  such  as  graphic  and  online  quizzes. 

This  project  develops  and  disseminates  instructional  software  architecture  that  will  serve  to 
integrate  empirical  investigation  and  the  experimental  method  into  the  undergraduate 
economics  curriculum.  Users  of  the  software,  which  will  be  available  over  the  internet,  will 
be  able  to  add  a  regular  laboratory  component  to  their  introductory  and  intermediate-level 
courses  in  economics.  The  laboratory  modules,  while  complete  in  themselves,  will  be 
structured  in  a  computer  technology  that  allows  faculty  to  easily  add  customizing  features, 
such  as  lecture  notes  and  graphic  images.  This  vdll  represent  the  first  extensive  use  of  the 
Microsoft  BackOffice  in  the  economics  curriculum.  Web  pages  use  a  SQL  Server  database  to 
generate  real  time  graded  question  and  answer  sequences,  with  responses  stored  in  and 
disseminated  from  the  database.  The  goal  of  the  proposed  project  is  to  bridge  the  theoretical 
and  empirical  aspects  of  economics  early  on  in  the  student’s  academic  career.  The  means  of 
achieving  this  goal  is  the  provision  of  computer  technology  that  makes  the  best  interactive 
and  accessibility  features  of  the  internet  available  to  faculty,  regardless  of  their  programming 
knowledge.  The  results  of  the  project  will  reach  non-economics  majors  who  take  introductory 
courses  in  the  field,  as  well  as  those  majoring  in  the  field.  Future  work  using  this  architecture 
could  extend  to  all  levels  of  the  undergraduate  major,  where  laboratory  modules  will  give 
students  a  better  feel  for  working  with  economic  data  and  for  the  ability  of  the  experimental 
method  to  reveal  the  basic  workings  of  the  market  mechanism.  Assessment  of  the  benefits  of 
using  this  technology  will  be  done  by  creating  an  assessment  test  administered  pre-  and  post¬ 
instruction  to  both  a  group  using  the  technology  and  a  control  group.  Electronic  auctions 
developed  may  enhance  acquisition  efficiencies  in  the  Department  of  Defense. 

The  workshop  demonstrated  how  student  users  interact  with  the  workbook,  how 
instructors  access  and  use  student  performance  reports,  and  how  content  authors  can  make 
changes.  In  addition  there  was  an  overview  description  of  the  software  architecture  of  the 
workbook  and  of  one  online  economics  experiment.  Finally,  there  was  a  discussion  of  the 
plans  of  future  development  and  assessment  methodology. 
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Just-in-Time  Teaching  (Workshop) 

Evelyn  T.  Patterson,  Ph.D. 

Department  of  Physics 

Abstract:  In  this  workshop  we  provided  a  hands-on  tutorial  on  Just-in-Time  Teaching 
(JiTT),  a  web-based  pedagogy  pioneered  as  a  collaborative  effort  of  physics  faculty  at 
USAFA  and  at  Indiana  University-Purdue  University  Indianapolis  (an  urban  campus 
serving  27,000  students).  JiTT  pedagogy  exploits  an  interaction  between  web-based 
study  and  an  active  learner  classroom.  Assignments  and  enrichment  materials  are 
delivered  to  the  students  over  the  web.  Students  respond  to  these  assignments  electronically.  These 
student  submissions  provide  immediate  feedback  to  the  instructors  concerning  the  state  of  the  class’ 
progress.  These  assignments,  due  in  the  morning  a  few  hours  before  class,  are  used  to  adjust  the  lesson 
content  and  activities  to  suit  the  students’  needs.  During  the  past  two  years  we  have  presented  these 
techniques  at  24  invited  workshops,  five  of  them  to  non-physics  audiences.  The  sponsor  organizations 
range  from  the  American  Association  of  Physics  Teachers  to  Project  Kaleidoscope  and  the  National 
Science  Foundation.  Educators  at  more  than  a  dozen  schools  are  currently  experimenting  with  this 
pedagogy  at  institutions  ranging  from  Harvard  University  to  the  University  of  Illinois  at  Urbana  to  The 
Nightingale-Bamford  School  in  Manhattan.  A  short  paper  detailing  the  essence  of  JiTT  can  be  found 
at  http://webphvsics.iupui.edu/JITT/cciitt.html 

Workshop  Audience 

The  intended  workshop  audience  was  faculty  currently  using  an  active  learner 
approach  to  teaching.  The  workshop  was  also  geared  toward  faculty  who  are  interested  in 
learning  about  the  various  ways  the  web  can  be  used  in  instruction  and  who  are  contemplating 
adopting  or  adapting  some  of  these  innovative  approaches  to  teaching.  No  special  skills  on  the 
part  of  the  workshop  participants  were  assumed,  and  no  technical  expertise  was  required. 

Workshop  Resources 

The  workshop  was  held  in  the  USAFA  Network  Classroom  Lab  (NCL),  where  each 
participant  had  individual  access  to  a  computer  workstation.  The  workstations  each  had 
internet  connections.  The  workshop  had  its  own  World  Wide  Web  homepage  at 
http://www.usafa.af.mil/dfp/phvsics/webphvsics/usafaiitt/ 

Workshop  Activities 

1 .  Participants  were  asked  to  complete  a  “preflight”  as  preparation  for  workshop 
involvement. 

2.  A  presentation  and  exploration  of  Just-in-Time  Teaching  (JiTT)  materials  developed  for 
our  introductory  physics  classes.  Rather  than  the  content,  we  emphasized  the  structure  of 
these  documents  and  the  rationale  behind  their  development,  focusing  on  their  relation  to 
classroom  instruction.  Thus  workshop  participants  contemplated  how  these  resources 
could  be  adapted  to  their  own  disciplines. 

3.  A  session  on  JiTT  pedagogy  (including  the  management  of  an  active  learner  classroom.) 

4.  Exploration  and  application  of  JiTT  templates,  including  exploratory  visits  to  resource 
sites  for  selected  other  disciplines.  Workshop  participants  were  encouraged  to  look  at 
these  materials  with  a  creatively  open  mind  and  imagine  how  these  resources  would 
enrich  their  own  teaching  initiatives. 

5.  A  discussion  of  implementation  details  and  technology  issues. 
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Workshop  Deliverables 

1 .  JiTT  HTML  templates  and  information  about  the  DFP  “Preflight  Editor”  that  is  used  to 
author,  deliver,  and  record  student  responses  to  preflights. 

2.  A  JiTT  handout  comprised  of  selected  excerpts  from  the  Just-in-Time  Teaching:  Blending 
Active  Learning  with  Web  Technology  book  (authors  G.  Novak,  E.  Patterson,  A.  Gavrin, 
and  W.  Christian)  to  be  published  in  March  of  1999  by  Prentice  Hall. 

3.  The  workshop  resources  remain  available  at  the  workshop  website, 
http://www.usafa.af.mil/dfp/phvsics/webphvsics/usafaiitt/ 

Description  of  JiTT  Pedagogy 

Since  its  inception  in  1995,  the  World  Wide  Web  technology  has  been  widely  adopted 
for  teaching  purposes.  Web  sites  serve  as  communication  hubs  for  student-teacher,  student- 
student  and  teacher-teacher  interactions.  Web-based  instructional  material  can  structure  the 
out-of-the-classroom  time  so  the  students  can  use  it  more  productively.  The  Just-in-Time 
Teaching  approach  blends  an  active  learner  classroom  with  the  around-the-clock 
communication  capability  of  the  World  Wide  Web.  Students  interact  with  one  another,  human 
instructors,  and  technology  in  ways  that  optimize  the  effectiveness  of  these  interactions. 

While  no  technology  can  match  the  benefits  of  an  expert  human  mentor  who  observes  the 
learning  activity  and  intervenes  as  needed,  there  are  aspects  of  learning  where  technology  can 
have  an  edge  over  a  human  instructor. 

Just-in-Time  Teaching  (JiTT)  is  a  teaching  and  learning  strategy  comprised  of  two 
elements:  classroom  activities  that  promote  active  learning  and  World  Wide  Web  resources 
that  are  used  to  enhance  the  classroom  component.  Many  industries  use  Just-in-Time  methods 
that  combine  high-speed  communications  and  rapid  distribution  systems  to  improve 
efficiency  and  flexibility.  Our  use  of  JiTT  is  analogous  in  many  ways.  We  combine  high¬ 
speed  communications  on  the  web  with  our  ability  to  rapidly  adjust  content.  This  makes  our 
classroom  activities  more  efficient  and  more  closely  tuned  to  our  students’  needs.  The 
essential  element  of  JiTT  is  feedback  between  the  web-based  and  classroom  activities. 

The  JiTT  web  materials  fall  into  several  broad  categories:  electronic  preparatory 
assignments,  enrichment  extra  credit  materials,  online  homework,  and  information  and 
communication.  The  heart  of  the  JiTT  strategy  is  in  the  use  of  the  electronic  preparatory 
assignments,  of  which  there  are  two  types:  the  WarmUp/Preflight  and  the  Puzzle.  These 
assignments  provide  an  introduction  and  a  conclusion  to  a  given  topic,  as  described  briefly 
below. 

The  WarmUp/Preflight  and  Puzzle  web  assignments  share  a  common  set  of 
characteristics: 

•  The  questions  asked  are  motivated  by  a  clear  set  of  learning  objectives 

•  The  web  assignments  introduce  students  to  the  technical  terms. 

•  The  questions  require  students  to  confront  their  previously  held  notions. 

•  The  questions  are  extendible. 

JiTT  classroom  lessons  are  built  around  student  answers  to  “Preflights”  (called 
“WarmUps”  at  most  institutions  other  than  USAFA).  Preflights  are  web-based  preparatory 
assignments  that  are  due  a  few  hours  before  class.  The  students  complete  these  assigiunents 
individually,  at  their  own  pace,  and  submit  them  electronically.  The  instructor  collects  the 
students’  electronic  submissions,  reads  them,  and  adjusts  and  organizes  the  classroom  lessons 
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in  response  to  the  student  submissions  “Just-in-Time.”  With  this  strategy,  the  instructor  learns 
what  the  students  already  understand,  what  they  find  difficult,  and  where  their  confusion  is, 
before  ever  beginning  class.  Typically,  during  the  lesson,  the  instructor-presents  a  selected 
(einonymous)  subset  of  the  student  responses  to  the  class,  weaving  them  into  the  class  as 
appropriate.  This  establishes  a  feedback  loop  between  the  classroom  and  the  web  that  is 
beneficial  to  students  and  instructors  alike.  Students  in  the  classroom  recognize  their  own 
wording,  both  correct  and  incorrect,  and  thus  become  engaged  as  part  of  the  feedback  loop. 
Discreet  in-class  critique  of  their  submissions  helps  improve  the  writing  skills  and  the 
construction  of  physics  knowledge.  It  is  quite  common  for  the  classroom  discussion  to 
continue  via  email  between  the  instructor  and  particular  students.  Paradoxically,  technology 
used  this  way  encourages  a  more  personal  and  intimate  bond  between  instructors  and 
students.  It  is  clear  from  course  evaluations  that  students  feel  part  of  a  team  working  on  a 
common  project. 

We  have  several  basic  objectives  in  constructing  a  Preflight  assignment: 

•  Set  the  firamework  for  the  physics  to  be  studied  in  practical  everyday  terms. 

•  Pique  the  student’s  curiosity;  generate  a  need  to  know. 

•  Relate  material  to  be  learned  to  material  already  studied. 

•  Give  the  student  a  sense  of  confidence. 

Having  done  the  assignment,  the  student  should  be  able  to  say,  “I  know  what  the 
lesson  will  be  about.  I  imderstand  the  scenario  and  I  understand  the  question.  I  have  some  idea 
of  the  answer  but  I  am  not  sure  and  I  don’t  know  all  the  details.” 

To  accomplish  the  above  objectives  we  present  each  question  in  a  context  that  can  be 
imderstood  without  reading  the  textbook.  However,  the  question  should  incorporate  the 
important  concepts  from  the  chapter.  We  typically  present  a  scenario  and  with  one  or  more 
questions  ask  the  student  to  predict  the  outcome.  It  is  important  that  the  question  be  simple 
and  straightforward  so  that  the  student  has  no  trouble  understanding  what  is  being  asked.  It  is 
equally  important  that  the  question  be  complex  enough  so  that  some  conceptual  understanding 
of  physics  is  necessary  to  provide  a  complete  answer.  When  the  underlying  physics  contains 
firequently  held  erroneous  preconceptions,  the  Preflight  question  is  constructed  so  as  to  flush 
out  these  preconceptions.  The  classroom  discussion  of  such  questions  must  be  explicit  enough 
to  persuade  the  students  to  give  up  the  preconceived  notions.  It  may  well  be  necessary  to 
bolster  the  case  with  a  well-planned  demonstration  or  activity.  Where  possible,  we  leave  some 
particulars  out,  to  be  supplied  by  the  student.  This  adds  variety,  which  makes  Ihe  classroom 
discussion  more  interesting. 

It  is  important  to  make  the  questions  good  discussion  starters.  It  is  not  uncommon  for 
students  in  JiTT  courses  to  be  talking  about  Preflight  questions  in  their  conversations  before 
class  begins.  The  best  Preflight  questions  engage  the  students  enough  that  they  are  actually 
thinking  about  the  questions  and  their  answers  prior  to  class.  Good  questions  fall  into  different 
categories  but  generally  share  one  or  more  of  the  following  features. 

•  are  attention  catchers 

•  refer  to  current  events 

•  are  just  ambiguous  enough  to  cause  the  students  to  really  pause  and  think 

•  explicitly  confront  problem  areas  identified  by  physics  education  research 

•  pick  up  on  a  thread  fi-om  a  very  recent  class  discussion  or  activity 

•  are  not  too  easy  or  too  hard 

•  include  at  least  some  (preferably  all)  nontechnical  phrasing 
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Another  feature  that  often  works  well  to  generate  discussion  and  facilitate 
understanding  deals  with  multiple  choice  questions.  We  sometimes  include  more  than  one 
choice  that  could  be  correct,  depending  on  the  assumptions  the  student  makes  about  the  setup. 
These  questions  then  are  deliberately  ambiguous  or  loosely  defined.  It  is  not  a  good  idea  to 
have  this  kind  of  question  unless  the  students  are  explicitly  told  that  “there  may  be  no  one 
correct  answer  to  this”  and  that  “the  purpose  of  this  question  is  to  elicit  discussion  in  class.” 
Furthermore,  students  should  not  be  awarded  course  points  based  on  correcmess  in  such  cases 
of  deliberately  vague  or  ambiguous  problems,  for  they  can  easily  become  disheartened.  The 
students  should  not  feel  that  they  are  being  penalized  for  having  been  “tricked”,  but  rather 
should  feel  that  the  question  has  brought  about  a  “teachable  moment”  that  has  helped  them  to 
solidify  their  xmder standing. 

Helping  students  recognize  that  their  “everyday”  understanding  of  words  and  ideas 
may  be  incorrect  or  incomplete  is  a  crucial  part  of  the  learning  process.  Furthermore, 
reminding  them  to  reconsider  and  reflect  on  actual  experiences  they  may  have  had,  and  asking 
them  to  confront  their  preconceived  ideas  about  the  way  nature  works,  helps  students  to 
construct  new  and  deeper  reasoning  and  understanding  [Fuller,  1982].  In  fact,  this  idea  is 
central  to  the  Learning  Cycle,  a  teaching  strategy  developed  by  physicist  Robert  Karplus 
[Karplus,  1977].  The  JiTT  web  component  helps  students  consider  their  everyday 
imderstanding  and  experiences,  realize  that  their  current  mental  constructs  are  incomplete,  and 
come  prepared  to  deal  with  them  in  the  classroom  where  they  are  guided  by  the  instructor. 

The  other  preparatory  web  assignment  is  the  Puzzle.  We  think  of  the  Preflight  and 
Puzzle  activities  loosely  as  bookends  for  a  given  topic,  with  the  Preflight  beginning  and  the 
Puzzle  concluding.  Since  the  puzzle  ends  a  topic,  it  is  subtler,  and  usually  requires  integration 
of  many  ideas.  Each  week  we  assign  one  Puzzle.  Typically,  it  is  a  single  question  that  may  be 
somewhat  vague  and  involves  several  concepts.  As  the  name  implies,  the  puzzle  is  a  physics 
scenario  with  an  extra  twist  that  requires  the  student  to  see  beyond  the  end-of-chapter 
formulas.  This  is  often  accomplished  by  further  challenging  commonly  held  misconceptions. 
The  Hestenes  Force  Concept  Inventory  questions  [Halloun  and  Hestenes,  1985]  and  Eric 
Mazur’s  peer  instruction  questions  [Mazur,  1996]  have  a  similar  character.  These  questions 
may  appear  trivial  to  a  physicist,  and  many  can  be  answered  without  calculations.  On  the 
other  hand,  they  usually  present  a  significant  challenge  to  the  beginning  student.  In  the 
classroom,  we  use  the  Puzzles  much  as  we  do  the  Preflights.  Based  on  excerpts  from  the 
students’  solutions,  we  lead  a  classroom  discussion  of  the  question  and  of  possible  variations 
and  extensions.  This  lends  itself  well  to  closing  a  topic  and  then  integrating  it  with  the  rest  of 
the  course  material.  Since  the  Puzzle  closes  the  topic,  we  expect  greater  precision  in  the 
answers,  and  we  evaluate  Puzzle  answers  more  rigorously.  Answers  should  be  technically 
correct  and  well  stated. 

In  addition  to  the  Preflight/WarmUp  and  puzzle  pages,  there  is  a  weekly  applications 
page  that  features  an  essay  on  some  topic,  related  but  not  central  to  the  course  material.  In  our 
physics  classes  this  page  is  titled,  “What's  Physics  Good  For?”  These  essays  differ  from  the 
“application”  essays  in  textbooks  in  important  ways.  First,  they  contain  numerous  links  to 
external  web  sites  containing  a  wealth  of  related  material.  Second,  they  always  end  with  extra 
credit  assignments,  which  induce  the  students  to  follow  the  web  links  and  thus  relate  the 
course  material  to  the  real  world. 
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Our  JiTT  courses  also  offer  information  and  communication  pages  that  contain  the 
electronic  edition  of  traditional  handout  material  such  as  the  syllabus  and  the  course  calendar 
as  well  as  the  electronic  version  of  the  course  bulletin  board. 

The  JiTT  web  material  is  designed  to  facilitate  an  effective  blend  of  human  and 
technological  resources.  The  web  assignments  encourage  students  to  prepare  for  the 
classroom  activity  where  the  instructors  will  provide  the  intellectual  mentoring.  The  key  to 
JiTT  is  to  use  web  communication  technology  to  prepare  both  the  students  and  the  instructors 
for  the  events  in  the  classroom.  Working  -with  the  JiTT  strategy  has  convinced  us  that  the 
web,  combined  with  live  teachers  in  the  classroom,  can  personalize  instruction  for  all  students 
and  make  a  real  difference  to  the  non-traditional  student. 

More  information  about  JiTT  is  available  via  the  JiTT  website, 
http://webphvsics.iupui.edu/iitt.htm.  and  will  be  available  shortly  through  the  Just-in-Time 
Teaching:  Blending  Active  Learning  with  Web  Technology  book  to  be  published  by  Prentice 
Hall  in  March  1999. 
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IT  in  the  Classroom;  Should  We  Throw  Away  the  Chalk?  (Panel) 

Margaret  E.  (Peg)  Halloran,  Ph.D. 

Deputy  Director,  Institute  for  Information  Technology  Applications 

Panel  Members;  Dr.  Evelyn  Patterson  (Dept  of  Physics),  Maj  Glenn  Hover  (Dept  of 

Behavioral  Sciences  and  Leadership) 

Abstract:  Academicians  are  both  attracted  to  and  repelled  by  the  thought  of  using  information 
technology  (IT)  as  a  teaching  tool.  Information  technology  can  be  used  to  enhance  classroom 
presentation  and  to  expand  the  classroom  beyond  traditional  time  and  space  limitations.  In  addition, 
interactive  study  guides  delivered  via  the  WWW  can  provide  a  mechanism  for  instantaneous  feedback 
to  the  student  on  their  progress.  However,  using  technology  can  also  limit  spontaneity  and  become  the 
focus  of  the  classroom  experience,  rather  than  a  teaching  tool.  Therefore,  the  purpose  of  this  panel  is 
to  discuss  the  advantages  and  disadvantages  of  using  technology  in  the  classroom,  and  to  discuss  when 
and  if  it  is  appropriate  to  “throw  away  the  chalk”. 

Academicians  are  both  attracted  to  and  repelled  by  the  thought  of  using  information 
technology  (IT)  as  a  teaching  tool.  The  attraction  is  the  value  of  being  able  to  communicate 
and  discuss  ideas  to  a  virtually  unlimited  number  of  people  in  a  greater  variety  of  ways 
without  geographical  and  time  barriers.  The  repellant  is  that  technology  will  prove  the 
undoing  of  academics.  It  has  been  argued  that  technology  uses  up  large  amounts  of  time  and 
money  producing  “canned”  presentations  and  that  multimedia  presentations  make  the  printed 
page  look  boring.  It  has  also  been  argued  that  multimedia  presentations  suppress  the 
imagination,  and  promote  the  decline  in  literacy  by  favoring  video  over  pictures,  and  pictures 
over  words.  In  this  panel,  we  discussed  some  advantages  and  disadvantages  of  using 
technology  as  a  teaching  tool. 

There  are  many  benefits  of  using  technology  to  enhance  the  classroom  learning 
environment.  Within  the  classroom,  colorful  visual  presentations  prepared  using  presentation 
development  software  (e.g.  PowerPoint),  can  be  used  to  provide  concise  outlines  for  students 
that  break  down  difficult  concepts  into  smaller  and  more  conceptually  manageable  ones. 
Computer-generated  animation  sequences  often  show  processes  that  are  difficult  to 
conceptualize  better  than  static  textbook  images  can  display  them.  In  addition,  document 
cameras  and  similar  projection  devices  can  project  instructor  demonstrations  on  a  screen  for 
easier  viewing  by  the  class.  Images  projected  from  a  microscope,  for  example,  are  easier  for  a 
group  to  view  when  projected  than  when  students  take  turns  looking  through  the  ocular  lens. 
These  images  can  often  be  photographed  and/or  stored  on  disc  for  later  reference. 

The  use  of  information  technology  can  expand  the  classroom.  Posting  assignments  and 
course  material  on  the  internet  allows  students  to  access  information  anywhere  and  at 
anytime,  which  provides  flexibility  for  busy  students  and  possibilities  for  making  up  missed 
classes.  Class  materials  can  be  linked  to  other  resources  from  other  departments  and 
institutions,  which  gives  interested  students  the  opportunity  to  further  explore  a  given  topic. 
Video  clips  and  animations  that  are  used  in  the  classroom  can  also  be  made  available  for 
students  to  review  outside  of  class  by  posting  them  to  a  website. 

Online  multimedia  study  guides  can  be  designed  to  address  different  learning  styles 
and  provide  a  mechanism  for  interactive  assignments  and  instantaneous  feedback.  Numerous 
studies  have  shown  that  most  students  retain  more  information  when  they  engage  in  active. 
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rather  than  passive,  learning  strategies.  In  addition,  online  practice  exams  give  students 
immediate  feedback  on  their  performance  and  their  level  of  understanding  the  information. 

An  individual  student’s  performance  can  be  automatically  tracked  by  the  instructor  and  can  be 
used  to  assess  which  students  might  require  extra  help  outside  of  class.  Tracking  can  also  be 
anonymous,  which  may  decrease  the  student’s  embarrassment  over  poor  performance  and 
encourage  participation  while  still  making  the  instructor  aware  of  difficult  concepts  and 
important  stumbling  blocks  by  the  class  as  a  whole. 

Videoteleconferencing  is  another  type  of  technology  that  can  expand  the  classroom  by 
providing  a  mechanism  for  guest  speakers  to  give  lectures  from  remote  sites  thousands  of 
miles  away.  This  is  beneficial  when  the  guest  lecturer  lacks  the  time  and  or  resources  to 
travel.  While  this  scenario  is  not  superior  to  the  personal  interactions  that  can  be  achieved  by 
an  in-class  visit,  it  does  open  up  new  possibilities  for  interactions  that  otherwise  might  not 
take  place. 

However,  the  use  of  technology  in  teaching  is  not  a  universal  panacea,  and  can  create 
difficulties.  Highly  polished  PowerPoint  presentations  may  look  nice,  but  the  instructor  risks 
“scripting  the  class”  which  is  not  always  compatible  with  Socratic  dialog.  The  instructor  may 
feel  compelled  to  stick  to  their  fine  tuned  outline  and  not  allow  deviations  from  their  outline 
in  response  to  questions.  In  addition,  these  presentations  are  often  delivered  with  the  lights 
turned  low,  and  giving  out  copies  of  the  slides  may  encourage  students  to  daydream  during 
class  since  they  know  they  will  get  the  material  in  a  hand  out. 

Easy  access  to  more  information  from  the  internet  doesn’t  mean  receiving  better 
information.  The  instructor  needs  to  provide  guidance  to  the  students  as  they  access  outside 
resources.  Currently,  there  are  no  standards  or  a  peer-review  process  for  ■website  content.  The 
easy  access  to  good  quality  websites  with  valid  content  is  therefore  diluted  by  the  plethora  of 
poor  quality  websites  with  questionable  content,  and  students  may  have  difficulty  in 
determining  the  quality  and  validity  of  these  resoiorces. 

Another  problem  with  using  technology  is  over  reliance  on  that  technology.  The 
WWW  may  be  inferior  to  traditional  textbooks  for  reading  assignments.  People  seem  to  be 
more  comfortable  reading  text  from  a  book,  than  off  of  a  computer  screen.  In  addition,  when 
students  are  assigned  to  readings  from  a  text,  they  generally  laiow  in  advance  the  length  of  the 
reading.  Simply  giving  students  the  website  address  does  not  convey  that  information,  which 
may  cause  them  to  underestimate  the  time  required  to  complete  an  assignment  and  contribute 
to  them  being  unprepared  for  class  or  an  examination. 

Once  you  rely  on  technology  to  accomplish  a  task,  the  technology  must  continue  to 
work.  Problems  as  small  as  printer  jams  and  burned  out  light  bulbs  can  ruin  even  the  best 
prepared  lesson.  Larger  problems  such  an  unstable  network  can  interfere  with  students’  ability 
to  access  their  assignments,  or  turn  in  homework.  This  ■will  lead  to  frustration  and  can  act  as 
an  impediment,  rather  than  an  enhancement  to  learning.  People  generally  must  be  able  to 
successfully  access  a  system  90  -  95%  of  the  time  to  have  confidence  in  its  reliability  and  to 
continue  to  access  the  system.  Since  most  internet  systems  rely  on  a  system  of  servers  and 
routers,  instability  of  one  part  of  the  system  can  have  exponential  negative  effects. 

Therefore,  technology  should  be  used  judiciously.  The  advantages  and  disadvantages 
of  using  technology  in  teaching  should  be  balanced  against  each  other.  It  is  also  important  to 
remember  that  there  are  two  different  users  -  one  who  is  teaching  and  the  other  who  is 
learning,  and  they  often  have  different  interface  and  design  needs.  However,  used  properly. 
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technology  is  one  more  tool  to  be  added  to  the  mix  of  teaching  tools  and  enhance,  rather  than 
detract,  from  learning. 
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Web-Based  Mathematica  Projects  for  Calculus 

Robert  J.  Clasen,  M.S.,  Captain,  USAF 
Department  of  Mathematical  Sciences 

Abstract:  We  incorporated  web-based  Mathematica  projects  into  our  Spring  1998  offering  of  a  first- 
semester  calculus  course.  This  paper  describes  our  projects  that  were  based  on  famous  battles  from 
history.  The  projects  are  provided  at  http://www.usafa.af.mil/dfins/projects/mathl41/spring98. 
Suggestions  for  adapting  our  projects  to  other  computer  algebra  systems  and  disciplines  are  included. 


At  the  United  States  Air  Force  Academy,  we  used  web-based  Mathematica  projects  in 
our  Calculus  I  course.  These  projects  were  intended  to  expose  students  to  the  use  of 
technology  as  a  problem  solving  tool  and  at  the  same  time  test  their  tmderstanding  of  basic 
calculus  concepts.  Each  project  was  based  on  a  famous  battle  from  history.  This  paper  gives 
a  description  of  the  projects,  which  are  located  at  the  following  web  address: 
http  ://www.usafa.af  mil/dfins/proi  ects/math  1 4 1  /spring98 . 

All  three  projects  had  the  same  basic  structure.  First,  an  overview  page  introduced  the 
project  to  the  students  and  described  what  they  were  expected  to  accomplish.  Then,  the 
students  read  a  short  summary  of  the  battle  and  answered  a  few  questions  about  the  battle. 
After  that,  the  students  used  Mathematica  to  answer  some  calculus  questions;  these 
mathematical  questions  were  related  to  some  aspect  of  the  battle.  Finally,  there  was  a  page 
containing  links  to  other  sites  that  had  related  information  to  the  battle,  so  those  who  were 
interested  could  easily  explore  the  topic  further. 

The  historical  portion  of  the  project  had  nothing  mathematical  about  it  at  all.  Its 
purpose  was  to  expose  the  students  to  a  bit  of  military  history  (an  important  goal  at  a  military 
academy)  and  also  make  the  project  a  little  more  interesting.  The  three  battles  used  as  themes 
for  these  projects  were  Gettysburg,  Agincourt,  and  Verdun.  The  students  would  read  a 
summary  of  the  battle  (not  too  long  —  about  three  or  four  screens  of  information)  and  then 
answer  some  questions  about  the  battle.  These  questions  ensured  the  students  actually  read 
the  description  of  the  battle,  but  they  only  accounted  for  about  20%  of  the  project's  total 
points.  Because  the  Air  Force  Academy  encourages  character  development  in  all  its  classes, 
these  questions  would  sometimes  be  ethical  in  nature.  For  example,  one  question  from  the 
Battle  of  Agincourt  asked  whether  or  not  it  was  appropriate  for  Henry  V  to  give  the  order  to 
kill  the  unarmed  French  prisoners  of  war. 

The  meat  of  each  project  was  contained  in  the  Mathematica  portion.  This  is  where  the 
students'  understanding  of  calculus  concepts  was  tested,  as  well  as  their  ability  to  use 
technology  as  a  problem  solving  tool.  The  math  questions  were  linked  to  the  battle  in  some 
way.  For  example,  a  key  weapon  in  the  English  defeat  of  the  French  at  Agincourt  was  the 
long  bow.  So,  the  students  were  given  a  function  (based  on  real  data)  that  gave  the  height  of 
an  arrow  shot  from  a  long  bow,  and  they  were  then  asked  to  plot  the  function,  interpret  the 
plot  of  the  height  to  give  information  about  the  velocity,  calculate  average  and  instantaneous 
velocities  of  the  arrow,  and  determine  how  long  the  arrow  was  in  the  air.  The  students  used 
Mathematica  to  answer  the  questions  and  turned  in  a  printout  of  the  Mathematica  notebook  to 
be  graded. 

These  projects  required  the  use  of  two  technologies.  The  first  was  the  use  of  the 
internet,  which  was  nice  but  not  essential  for  a  math  course.  It  made  the  project  more 
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appealing  through  the  use  of  pictures  and  multimedia  (we  used  a  video  clip  of  actual  footage 
from  the  trenches  of  Verdun),  and  the  page  of  related  web  links  made  if  very  easy  for  those 
who  wanted  to  explore  further  to  do  so.  However,  the  historical  content  could  also  have  been 
conveyed  through  the  use  of  a  handout.  For  those  who  can  easily  create  and  post  web  pages, 
though,  it  seems  worth  the  effort  to  use  the  internet  since  today's  students  are  great  web 
surfers  and  love  flashy  pages. 

The  second  use  of  technology  was  Mathematica.  We  wanted  to  show  the  students 
how  computer  algebra  systems  can  make  their  lives  easier  by  eliminating  the  tediousness  of 
long  calculations  and  by  easily  plotting  functions  to  help  visually  understand  a  problem. 

Since  the  vast  majority  of  our  students  are  technical  majors,  we  want  them  to  use  technology 
early  and  often  throughout  their  academic  career  here. 

The  feedback  from  the  students  about  these  projects  has  been  very  positive.  Though 
many  complain  about  Mathematica  being  picky  on  syntax,  they  seem  to  realize  they  will  be 
using  technology  throughout  their  careers  and  that  they  need  to  be  comfortable  with  its  use. 
They  also  seemed  to  enjoy  learning  about  the  battles  while  at  the  same  time  learning  math 
concepts. 
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Development  and  Use  of  Multimedia  Courseware  using  HTML 
Yumiko  Guajardo,  M.A. 

C 1 C  Matthew  S .  Komatsu 
Department  of  Foreign  Languages 

Abstract:  This  paper  focuses  on  the  development  and  use  of  multimedia  courseware  using  HTML 
and  various  audiovisual  devices  in  developing  materials  that  foster  communicative  and  cooperative 
learning  activities  designed  to  encourage  student  creativity  and  enhance  the  learning  experience.  It 
demonstrates  the  advantages  of  instantaneous  feedback  through  the  use  of  computer  programs,  and 
offers  examples  of  student-teacher  collaboration  in  the  development  and  use  of  these  materials  and 
activities.  Specific  demonstrations  focus  on  content-based  instruction  in  Japanese,  though  the  basic 
principles  may  be  applied  to  the  study  and  teaching  of  all  languages. 

Introduction 

Due  to  the  continuous  and  rapid  development  of  technology,  our-society  is  undergoing 
radical  transformations  in  every  conceivable  field  of  human  endeavor.  In  the  field  of 
instructional  technology,  the  increasing  use  of  media  centers  and  computer  assisted  language 
laboratories  has  in  many  cases  outpaced  the  integration  of  that  technology  with  classroom 
instruction.  Indeed,  many  language  instructors  find  the  prospects  of  using  this  novel 
technology  to  be  both  daunting  and  even  alienating.  This  has  been  particularly  so  in  the  case 
of  teachers  of  the  less  commonly  taught  languages  who  have  tended  to  rely  heavily  on 
traditional  means  of  instruction  or  allowed  their  colleagues  in  the  more  commonly  taught 
languages  to  break  new  ground  before  timidly  or  reluctantly  exploring  the  possibilities 
presented  by  the  new  forms  of  electronic  media. 

According  to  a  recent  study  in  Texas  (Moore,  Morales  &  Carel,  1998),  most  teachers 
will  make  considerable  use  of  traditional  video  materials,  but  few  instructors  will  venture  into 
the  realm  of  interactive  media  such  as  CD-ROMs,  videodiscs,  and  the  World  Wide  Web  even 
in  the  teaching  of  the  target  language's  culture.  Teachers  are  aware  that  computer  assisted 
language  learning  has  definite  advantages  when  it  comes  to  combining  text,  graphics,  sound, 
animation  and  video,  yet  teachers  continue  to  rely  heavily  on  textbooks.  According  to  the 
Moore  study,  among  the  small  percentage  of  teachers  who  actually  did  try  to  incorporate 
computer  technology  into  their  classroom  instruction,  more  experienced  teachers  (with  6-10 
years  teaching  experience)  tended  to  use  the  new  technology  more  than  less  experienced 
teachers  (with  less  than  5  years  experience).  In  other  words,  more  experienced  teachers, 
despite  being  somewhat  wary  of  and  intimidated  by  the  new  technology,  made  a  conscious 
effort  to  learn  and  try  to  integrate  computer  technology  into  their  classrooms,  than  did 
younger  and  less  experienced  instructors  who  often  were  more  familiar  with  the  new  media. 

It  is  clear  then  that  a  number  of  experienced  educators  are  very  much  aware  of  the 
potential  that  the  new  technology  holds  in  enhancing  the  learning  process.  It  is  also  clear  that 
enthusiasm  for  the  new  media  is  compromised  by  several  factors,  including  (1)  the  inability  of 
many  schools  to  adequately  equip  themselves  -with  the  hardware  and  resources  required  by 
advanced  technology;  (2)  Ae  dearth  of  satisfactory  software  programs,  CD-ROMS  and 
videodiscs  on  the  commercial  market;  (3)  the  cost  of  these  programs,  particularly  when  so  few 
address  the  needs  of  the  curriculum;  and  (4)  the  time  constraints  required  to  compile  the 
scattered  resources  of  the  WWW  in  order  to  make  them  useful  in  day  to  day  classroom 
instruction. 
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Koppany  (1995)  notes  —  particularly  in  the  less  commonly  taught  languages  —  that 
with  few  exceptions  there  is  a  shortage  of  usable  foreign  language  software.  Most  of  the 
available  language  learning  software  lacks  relevance,  currency,  adaptability,  authenticity  and 
communicative  interaction,  all  essentials  in  teaching  for  proficiency.  Teachers  are  left  to 
struggle  with  the  issues  involved  in  the  proper  integration  of  computer  technology  into  the 
curriculum.  As  Koppany  states,  "software  producers  continue  to  chum  out  FL  tutorials  of 
dubious  value,  while  teachers  continue  to  shake  their  heads  in  disappointment." 

Rationale  of  the  Project 

The  recently  published  National  Standards  for  Foreign  Language  Learning  (1996) 
emphasizes  the  key  "5C"  concept  in  language  teaching:  Communication,  Cultures, 
Connections,  Comparisons  &  Communities.  Specifically,  the  National  Standards  advocate  a 
close  coupling  of  the  target  language  with  its  attendant  culture,  enabling  the  language  learner 
to  compare  and  contrast  his/her  own  language  and  cultural  background  with  that  of  the 
language  and  culture  he/she  is  attempting  to  learn.  The  Standards  also  advocate  an  integration 
of  language  learning  with  information  and  knowledge  from  other  academic  and  practical 
disciplines,  requiring  teachers  to  present  geographical,  historical,  socioeconomic,  and  cultural 
materials  along  with  materials  used  in  day  to  day  language  instmction. 

No  single  language  textbook  can  address  this  rather  holistic  approach.  Consequently, 
the  Department  of  Foreign  Languages  here  at  the  United  States  Air  Force  Academy  has 
sought  to  create  language  teaching  materials  that  incorporate  a  cultural  context.  We  linked  a 
wide  range  of  sociocultural  topics  and  information  related  to  specific  areas  of  our  curriculum 
and  made  it  readily  available  to  students  over  the  Internet.  Currently,  the  Japanese  Language 
Program  has  four  categories  of  programs  to  supplement  language  instmction:  (1)  conventional 
language  lessons,  namely  Listening  Comprehension  Exercises  and  Reading  Comprehension 
Exercises;  (2)  Culture  Learning  Lessons,  including  geography,  history,  society  &  culture;  (3) 
Literature;  and  (4)  Government  and  Constitution  of  Japan,  including  the  armed  forces  and 
diplomatic  relations. 

The  production  of  this  courseware  is  based  on  HTML,  a  very  basic  and  simple 
programming  language.  While  the  program  lacks  sophisticated  bells  and  whistles,  i.e., 
animation  and/or  3D  images,  it  does  contain  photographs,  drawings,  sound,  and  scanned 
Japanese  characters.  The  scanned  images  of  the  characters  enables  anyone  to  view  them  on 
the  internet  without  recoiorse  to  Japanese  software,  overcoming  obstacles  caused  by  the 
variety  of  different  approaches  to  represent  non-alphabetic  writing  that  appear  on  different 
computer  platforms. 

The  Importance  of  Collaboration  with  Students 

One  reason  why  commercial  software  often  fails  to  address  the  needs  of  teachers  lies 
in  the  fact  that,  more  often  than  not,  it  is  produced  by  computer  technology  specialists  who 
focus  more  on  the  medium  than  on  the  content  of  their  products.  Also,  most  software 
developers  seek  to  gear  their  products  to  as  wide  an  audience  as  possible.  While 
understandable  firom  a  purely  marketing  perspective,  all  too  often  the  generic  product  ignores 
the  needs  of  specific  communities  of  learners. 

Good  teachers  know  that  effective  transmission  of  subject  material  occurs  when  it  is 
relevant  to  a  student's  background  and  interests.  Presenting  materials  from  the  teacher's 
perspective  (or  that  of  a  remote  software  engineer)  and  forcing  students  to  passively  receive 


63 


their  knowledge  is  not  likely  to  be  as  successful  as  getting  the  student  to  actively  participate  in 
the  acquisition  of  knowledge.  Supposedly,  this  is  the  whole  point  of  the  interactive  media,  but 
experience  shows  that  even  the  best  interactive  programs  will  be  conduits  of  passive  learning 
unless  the  teacher  provides  the  structure  for  interactive  learning  to  take  place.  To  do  this 
effectively,  teachers  must  interact  with  their  students.  They  must  delve  into  the  backgrotmds, 
interests,  and  aspirations  of  their  particular  students. 

Quite  often  the  computer  skills  of  students  are  more  advanced  than  those  of  their 
instructors,  and  it  is  here  where  students  and  teacher  can  cooperate  and  collaborate  in  the 
production  of  materials  that  are  not  only  relevant  to  the  particular  needs  and  interests  of  the 
students  but  are  also  relevant  to  the  course  and  an  excellent  means  of  exploring  different 
aspects  of  the  subject  material.  Naturally,  cooperative  and  collaborative  learning  need  not  and 
should  not  be  derived  exclusively  from  computer  generated  materials,  but  collaborative 
exercises  in  this  area  may  go  a  long  way  towards  providing  the  material  the  instructor  -wishes 
to  cover  while  also  providing  a  pro-active  means  for  the  students  to  participate  in  their  o-wn 
acquisition  of  knowledge. 

Granted,  time  and  resource  constraints  may  limit  the  effective  participation  of  all 
students,  particularly  in  large  classrooms,  in  the  production  of  software  material,  but  even 
collaborative  efforts  with  a  select  number  of  students  -will  benefit  the  class  as  a  whole  as  the 
selected  students  take  on  the  role  of  peer  instructors  and  demonstrate  what  they  have  learned 
through  the  medium  of  their  simple  products  before  the  class.  Or  the  teacher  can  assign 
cooperative  mini-projects  to  groups  of  students  and  compile  each  of  the  smaller  projects  into  a 
cumulative  whole. 

Such  projects  often  take  advantage  of  the  -wide  variety  of  backgrounds  that  students 
can  bring  to  the  study  of  specific  subject  material.  For  instance,  in  the  Japanese  history  class 
here  at  USAFA,  students  with  strong  backgrounds  in  western  and  world  history,  philosophy, 
or  literature  pro-vided  ftesh  perspectives  and  interpretations  about  the  relevance  of  Japanese 
history  for  themselves  and  the  class  as  a  whole,  ''^le  this  sort  of  interaction  can  be  achieved 
through  simple  class  discussion,  structuring  ideas  and  facts  into  a  coherent  form  through  the 
electronic  media  lends  itself  to  more  articulate  forms  of  expression  and  a  heightened 
appreciation  of  the  material.  In  addition,  the  concrete  product  which  can  be  used  as  an 
example  in  subsequent  classes,  is  far  less  transitory  than  the  mere  utterances  that  come  and  go 
in  a  mere  discussion  of  the  subject.  Consequently,  I  have  been  encouraged  to  work  with 
students  whose  majors  are  as  diverse  as  Legal  Studies,  Philosophy,  History,  and  other  social 
sciences. 

Conclusion 

All  of  the  courseware  that  I  and  my  students  have  put  together  during  the  past  two 
semesters  can  be  viewed  at  USAFA's  Department  of  Foreign  Languages  intranet  website 
(http://intraweb.usafa.af.mil/dfftlanguages/Japanese/index.htm)  as  well  as  through  the 
Colorado  World  Language  Improvement  Program's  site  which  is  linked  to  our  departmental 
site  on  the  internet  (htq)://www.usafa.af.mil/dfftclip/japanese/index.htm). 

These  collaborative  efforts  -with  my  students  have  pro-vided  them  -with  a  deeper 
understanding  of  the  subject  matter  as  well  as  providing  me,  as  their  instructor,  -with  material 
and  perspectives  that  will  appeal  to  and  inform  students  of  subsequent  classes.  The  use  of  the 
HTML  programming  language  provides  a  simple  means  of  creating  student-generated 
webware  that  can  be  learned  quickly  and  easily.  It  is  a  cost  effective  means  for  creating 


64 


relevant  supplementary  material  that  can  be  geared  to  the  specifics  of  classroom  instruction 
and  is  limited  only  by  the  amount  of  effort  and  imagination  the  instructor  and  students  are 
willing  to  invest.  Projects  of  this  nature  can  go  a  long  way  towards  addressing  the  specific 
needs  and  requirements  of  a  particular  subject  matter,  curriculum,  or  classroom. 
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A  Primer  to  Interactive  WWW-Based  Tutorials 


Margaret  E.  (Peg)  Halloran,  Ph.D. 

Deputy  Director,  Institute  for  Information  Technology  Applications 

Abstract:  One  of  the  best  features  of  the  WWW  is  the  ability  to  construct  interactive  text  and  graphic- 
based  pages.  There  are  many  approaches  to  obtain  different  levels  of  interactivity  within  a  web-based 
document.  Some  approaches,  such  as  JAVA  or  cgi-scripts  written  in  PERL,  require  sophisticated 
programming  abilities.  Other  approaches  use  simpler  HTML  codes,  which  can  be  modified  with 
JavaScript  for  greater  flexibility.  The  purpose  of  this  paper  is  to  provide  some  examples  of  different 
programming  tools  that  can  be  used  to  incorporate  interactivity  into  web  page  design. 

One  of  the  best  features  of  the  WWW  is  the  ability  to  construct  interactive  text  and 
graphic-based  pages.  These  interactive  pages  give  students  the  chance  to  input  quiz  answers, 
alter  the  parameters  of  a  computer  simulation  or  model,  and  receive  immediate  feedback  on 
their  work.  Although  some  of  these  designs  require  sophisticated  programming  ability  and/or 
web  development  software,  other  interactive  web  pages  can  be  created  by  incorporating  short 
segments  of  programming  code  or  “scripts”  into  the  HTML  document  by  someone  with  little 
or  no  programming  experience.  The  purpose  of  this  paper  is  to  give  a  brief  overview  of 
interactive  web  page  programming  from  the  simple  to  the  complex,  citing  examples  from  both 
the  sciences  and  the  humanities. 

One  method  of  developing  interactive  web  pages  is  to  use  PERL,  a  standard  CGI 
(common  gateway  interface)  programming  language,  which  is  stored  and  executed  by  the  web 
server.  Quiz  programs  written  in  PERL  can  give  immediate  feedback  to  the  student  on 
practice  or  graded  exams.  After  the  student  selects  their  choices,  the  PERL  script  grades  the 
quiz,  sends  the  students  responses  to  a  file  and  gives  the  student  feedback  as  to  which 
questions  they  missed.  Multiple  choice  quizzes,  such  as  the  General  Biology  test  bank  at  the 
University  of  Colorado  (IittD://www.colorado.edu/itrc/quiz/halloran/EPOB  121 0777. 1  .htmlJ 
can  be  modified  to  include  a  variety  of  options  including  correct  answers  to  missed  questions 
and  supplemental  material  for  further  study.  This  program  can  also  be  used  for  true-false  type 
questions. 

Programs  written  in  PERL  are  not  limited  to  multiple  choice  and  true-false  types  of 
exam  questions.  Students  can  also  get  feedback  on  short  answer  questions  and  questions 
based  on  diagrams  and  illustrations.  In  the  online  tutorial  to  accompany  the  Power  of  Logic 
philosophy  textbook,  students  get  the  opportunity  to  practice  complex  mathematical 
probability  problems  (http://www.poweroflogic.com/cgi- 

pol/Arith/arithmetic.cgi?exercise=l  I.IA)  and  are  asked  to  solve  logic  puzzles  using  Venn 
diagrams  (http://www.poweroflogic.com/cgi-pol/Venn/venn.cgi?exercise=5.2).  For  these 
tutorials,  students  are  only  told  that  their  answers  are  right  or  wrong,  providing  them  with 
imlimited  tries  to  solve  the  problems. 

One  advantage  to  using  a  server-side  language  such  as  PERL,  is  that  it  is  easy  to  track 
an  individual  or  group  of  students’  performance  by  having  output  from  the  program  generated 
to  a  file.  Instructors  can  access  these  files  and  assess  how  many  students  are  answering  the 
questions  correctly.  Depending  on  the  formatting  of  the  output  files,  quizzes  can  be  set  up  to 
record  answers  for  a  whole  class  anonymously,  or  an  individual  student’s  responses  to  each  of 
the  questions  can  be  tracked.  This  output  can  also  be  made  available  to  the  students  and 
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incorporated  into  an  online  grade  book  with  other  course  scores 
(http://snaefell.tamu.edu/~colin/Logic/lookmeup.html). 

JAVA  is  another  language  with  a  lot  of  versatility  and  is  useful  for  programming 
complex  interactive  simulations  and  models.  JAVA  programs  or  “Applets”  are  downloaded 
from  the  server  onto  the  client’s  browser.  A  good  example  of  a  JAVA  Applet  is  the  Evolve-IT 
simulation  nittD://\vww.cdl.edu/EvolveIT/Simulation/SimApplet.html').  In  this  simulation, 
students  manipulate  the  parameters  of  two  groups  of  birds  living  on  isolated  islands  and 
submit  these  parameters  to  the  server,  which  subsequently  generates  population  survival 
curves.  There  are  no  right  or  wrong  answers  to  be  graded,  but  rather  side  by  side  comparisons 
of  the  two  models  generated  from  students’  inputs  are  displayed. 

Constructing  PERL  and  JAVA  scripts  to  develop  sophisticated  interactive  tutorials 
both  require  a  considerable  amount  of  programming  knowledge.  However,  some  simple  tasks 
such  as  interactive  image  maps  can  be  created  using  basic  HTML.  For  example,  when  a 
person  mouse  clicks  on  a  state  on  the  United  States  Squirrel  Finder  map 
(http://spot.colorado.edu/~halloran/sqfinder.html').  a  listing  of  the  squirrel  species  found 
within  that  state  are  displayed. 

Image  maps  can  also  be  enhanced  using  JavaScript,  which  is  another  programming 
language  that  is  run  on  the  client  side  or  browser.  JavaScript  enabled  maps  are  slightly  more 
complicated  to  compose  than  maps  done  in  HTML,  yet  allow  for  a  greater  amount  of 
interactivity.  For  example  in  a  microscope  tutorial  image  map 
(http:// spot. Colorado .edu/~haIloran/scope .htmL.  placing  the  mouse  over  part  of  the 
microscope  will  have  a  different  effect  than  clicking  on  that  same  part.  JavaScript  is  not  as 
powerful  as  server  side  scripting,  but  can  be  useful  for  simple  routines.  In  addition,  unlike 
PERL  or  JAVA  Applets,  the  source  code  can  be  easily  viewed.  Therefore  if  you  see 
something  on  the  web  that  you  like,  you  can  easily  incorporate  it  into  a  new  page. 

There  is  a  lot  of  overlap  among  languages  in  their  usefulness  for  designing  interactive 
web  pages.  For  example,  multiple  choice  quizzes  can  be  written  in  both  PERL  and  JavaScript, 
with  slightly  different  features.  Also,  it  is  often  possible  to  combine  scripts  from  different 
languages  to  achieve  the  desired  effect.  When  selecting  a  language,  you  must  take  into 
account  the  types  of  interactivity  on  the  page,  your  programming  skills,  and  whether  the 
outputs  need  to  be  written  to  a  file.  The  rest  of  the  interactive  page  design  is  dependent  upon 
your  imagination. 
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Implementation  and  Assessment  of  Flexible  Learning  in  Geography  210 

Brian  E.  Heckman,  Ph.D. 

Department  of  Economics  and  Geography  - 

Abstract:  One  of  the  major  experiments  of  the  Geography  210  revision  program  is  the  design, 
implementation,  and  assessment  of  flexible  learning  lessons.  Of  the  17  sections  of  Geography  210  for 
the  Spring  1999  semester,  two  were  designed  as  experimental  or  "Z"  sections.  The  two  instructors 
design,  produce,  and  implement  one  or  more  lessons  using  a  course  development  tool  designed  to 
assist  busy  faculty  with  multimedia  development.  The  instructors  use  the  Course  Materials  Toolkit  as 
lesson  templates  for  instructional  development.  Once  complete,  students  work  on  the  lessons  either 
independently  or  in  the  usual  classroom  setting  where  the  instructor  serves  as  a  mentor.  Following 
implementation,  an  assessment  team  measures  the  effectiveness  and  efficiency  of  these  lessons. 

[Manuscript  unavailable] 
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Personal  Computer  Web  Site  Support  for  Instruction 

James  C.  Miller,  Ph.D.,  CPE 
Director,  Human-Environmental  Research  Center 

Abstract:  Descriptive  data  are  reported  about  the  use  of  a  web  site  on  an  instructor's  personal 
computer  for  information  dissemination  to  students.  Simple  descriptive  data  were  collected 
concerning  web  site  hit  rates  as  functions  of  sequential  lessons  and  days.  In  addition,  subjective 
ratings  of  site  usefulness  and  quality  were  acquired  approximately  halfway  through  the  semester  and 
near  the  end  of  the  semester,  four  days  before  the  final  exam.  The  project  produced  mixed  reviews  of 
the  web  site  from  the  users,  an  easily  interpretable  hit  rate  pattern  and  one  practical  lesson  about  file 
names  and  formats.  For  an  instructor  versed  in  HTML  or  the  Netscape  Composer®  or  Microsoft 
Frontpage®  editing  environments,  the  personal  web  server  concept  will  prove  to  be  a  useful  tool  for 
the  dissemination  of  information  to  cadets  and  other  students.  This  conclusion  must  be  caveated  as 
follows:  The  users  must  all  have  access  to  the  site  and  be  able  to  use  web  browser  software,  and  the 
network  must  function  reliably. 

New  technologies  provide  added  approaches,  beyond  traditional  approaches,  for 
making  instructional  information  available  to  students.  Descriptive  data  are  reported  here 
about  one  approach:  the  use  of  a  web  site  on  an  instructor's  personal  computer  for 
information  dissemination  to  students.  The  rationale  for  the  study  included  several 
considerations.  First,  our  instructors  and  students  were  linked  by  a  local  area  network  (LAN), 
to  which  all  were  connected  directly.  The  direct  connections  provided  high  speed 
transmission  of  data.  Second,  there  was  already  a  large  amount  of  communication  between 
instructors  and  students  by  electronic  mail  (e-mail)  on  this  network,  and  students  reportedly 
used  the  LAN  extensively  to  "surf'  the  World  Wide  Web.  Thus,  there  was  an  assumption  that 
the  students  had  at  least  basic  computer  skills  relevant  to  this  approach.  Finally,  there  was  an 
assumption  that  the  idea  of  accessing  instructional  data  by  web  "browsing"  would  be 
attractive  to  many  students. 

Methods 

Experimental  Design.  No  experimental  manipulations  were  used.  Simple  descriptive 
data  were  collected  concerning  web  site  hit  rates  as  functions  of  sequential  lessons  and  days. 

In  addition,  subjective  ratings  of  site  usefulness  and  quality  were  acquired  approximately 
halfway  through  the  semester  (25*^  lesson  out  of  42  lessons)  and  near  the  end  of  the  semester 
(lesson  42),  four  days  before  the  final  exam.  The  two  sets  of  ratings  were  subjected  to  simple, 
non-parametric  analyses  structured  to  detect  differences  that  might  have  occurred  as  a 
function  of  elapsed  time. 

Technology.  The  site  was  placed  on  an  Intel  Pentium®-based,  133-mhz  computer 
with  32  MB  RAM  and  its  own  Internet  protocol  (IP)  numeric  address.  The  web  site  capability 
was  supported  by  Microsoft  Personal  Web  Server®  software,  supporting  the  hypertext 
transport  protocol  (HTTP).  The  site's  one  "web"  page  was  prepared  initially  using  Netscape 
Composer®  software,  and  edited  later  in  its  raw  hypertext  markup  language  (HTML)  using 
the  word  processing  application  program,  Microsoft  Word  97®.  The  site  was  available  only 
on  the  LAN,  and  not  from  the  World  Wide  Web. 

The  contents  of  the  site  were  as  follows.  A  single  HTML  page,  the  home  page, 
included  a  welcoming  statement  that  identified  it  as  the  home  page  for  the  applicable  class. 
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This  confirmed  for  the  student  his  or  her  web  location.  The  page  also  included  an  attractive 
image  relevant  to  the  course  and  a  series  of  hypertext  links.  Most  of  the  links  were  to  word 
processor  documents  at  the  site.  These  documents  were  (1)  lists  of  course  objectives  for  each 
of  the  several  blocks  of  the  course  and  (2)  the  instructor's  extensive  lesson  plan  notes, 
organized  by  lesson  objective.  These  notes  included  information  from  the  course  texts  and 
additional  information  provided  by  the  instructor.  Because  the  course  was  a  "new 
preparation,"  using  a  new  textbook,  the  lesson  notes  only  became  available  on  the  site  shortly 
before  the  day  of  the  given  lesson. 

There  was  also  a  question  and  answer  page  at  the  site.  This  page  was  used  primarily 
to  display  instructor  discussions  about  questions  that  had  posed  difficulties  on  mid-term 
exams  during  the  course.  The  home  page  also  contained  and  displayed  a  "hits"  counter,  based 
upon  public  domain  common  gateway  interface  (CGI)  software.  The  counter  did  not  record 
the  use  of  the  word  processor  to  edit  and  inspect  the  home  page.  This  reduced  sharply  the 
number  of  non-data  webmaster  hits  counted  at  the  site.  The  other  home-page  contents 
included  hypertext  links  for  e-mail  to  each  of  the  two  instructors  in  the  course  and  information 
about  the  instructors'  office  and  telephone  numbers.  There  was  also  a  hypertext  link  to  a 
relevant,  interesting  external  site. 

Subjects.  The  subjects  were  63  cadets  at  the  Air  Force  Academy.  Primarily,  they 
were  in  their  third  year  at  the  Academy  and  taking  their  first  or  second  majors  class  as  Human 
Factors  Engineering  majors  in  the  Department  of  Behavioral  Sciences  and  Leadership  during 
the  fall  semester  of  1998.  The  class,  itself,  was  named  Introduction  to  Human  Factors  (BS 
373).  The  cadets  were  grouped  into  four  sections.  The  instructor  who  was  the  investigator 
and  web  site  developer  taught  one  of  these  sections,  and  the  instructor  who  was  the  course 
director  taught  the  other  three  sections. 

Procedures.  The  site  was  introduced  to  the  cadets  during  lesson  4.  They  were  given 
its  IP  address  and  told  of  its  contents.  This  information  was  repeated  in  response  to  cadet 
questions  throughout  the  remainder  of  the  course.  Web  site  hits  were  recorded  anonymously 
and  automatically,  then  reviewed  and  entered  manually  day  by  day  into  a  spreadsheet  by  the 
investigator.  Webmaster  hits  were  subtracted  from  the  total  number  of  hits  prior  to  data 
reduction  and  analysis.  The  simple  questionnaire  (attached)  was  administered  twice, 
anonymously,  during  class  time.  The  questionnaire  asked  for  the  cadet's  estimated  number  of 
hits  and  three  ratings:  the  usefulness  of  the  site,  the  quality  of  the  information  on  the  site  and 
the  usefulness  of  the  Intranet  web  server  concept.  Ratings  were  on  a  9-point  scale  anchored 
with  the  following  terms/phrases:  Excellent  (9),  Good  (7),  Borderline  (5),  Not  very  good  (3), 
and  Extremely  poor  (1)  (Babbitt  &  Nystrom,  1989). 

Due  to  an  inherent  unreliability  of  the  LAN  that  was  peculiar  to  this  semester,  there 
were  many  anecdotal  reports  by  cadets  of  an  inability  to  access  the  web  site.  Thus,  after  the 
first  graded  review  (GR;  midterm  exam),  the  lesson  plan  notes  present  on  the  web  site  were 
made  available  to  the  cadets  using  hard  copies,  an  intranet  hard  drive  and  e-mail  in  addition  to 
the  web  site. 

Results 

The  project  produced  mixed  reviews  of  the  web  site  from  the  users,  an  easily  interpretable  hit 
rate  pattern  and  one  practical  lesson  about  file  names  and  formats,. 

The  first  use  of  the  questionnaire  generated  54  responses.  The  cadets'  estimated 
number  of  hits  overestimated  by  1 1.8%  the  actual  number  of  hits  recorded  at  the  site.  The 
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mode  of  the  distribution  of  ratings  for  the  usefulness  of  the  site  was  7  ("Good"),  ranging  from 
9  ("Excellent")  to  3  ("Not  very  good").  The  mode  of  the  distribution  of  ratings  for  site 
information  quality  was  7  ("Good"),  ranging  from  9  to  6  (between  "Good"  and  "Borderline"). 
The  mode  of  the  distribution  of  ratings  for  the  Intranet  server  concept  was  7,  ranging  from  9 
to  1  ("Extremely  poor"). 

The  second  use  of  the  questionnaire  generated  56  responses.  The  cadets'  estimated 
number  of  hits  overestimated  by  only  5.0%  the  actual  number  of  hits  recorded  at  the  site.  The 
mode  of  the  distribution  of  ratings  for  the  usefulness  of  the  site  was,  again,  7  with  a  secondary 
mode  appearing  at  9  and  a  range  from  9  to  1 .  The  mode  of  the  distribution  of  ratings  for  site 
information  quality  was  7,  ranging  from  9  to  5  ("Borderline").  The  mode  and  range  of  the 
distribution  of  ratings  for  the  Intranet  server  concept  were  the  same  as  recorded  at  Lesson  25. 

A  nonparametric  test  was  used,  post  hoc,  to  compare  ratings  from  lesson  25  to  ratings 
from  lesson  42.  The  test  for  k  independent  samples  (Siegel,  1956)  was  adapted  for  use.  It 
was  applied  only  to  ratings  of  6  and  greater  so  that  expected  frequencies  would  be  adequately 
large.  The  number  of  degrees  of  freedom  for  the  test  (df  =  3)  were  halved  (df  =  1)  to  account 
for  the  lack  of  independent  samples.  (Note  that,  since  the  questionnaire  was  anonymous, 
there  was  no  record  of  the  repeated  measures  across  lessons  25  and  42.)  The  results  of  the 
tests  indicated  no  reliable  change  for  perceptions  of  site  usefulness  (A^  =  1.526,  p  <  0.30)  or 
quality  (A^  =  2.414,  p  <  0.20)  across  lessons,  but  a  significant  change  for  the  rating  of  the 
server  concept  (A^  =  4.260,  p  <  0.05).  The  rating  of  the  server  concept  declined  reliably  from 
lesson  25  to  lesson  42. 

Figure  5  shows  the  site  hit  rate  as  a  function  of  lesson  number  (lessons  1  through  42 
and  the  final  exam).  Figure  6  shows  the  site  hit  rate  as  a  function  of  calendar  date.  Modes  in 
the  distribution  of  hits  per  lesson  are  evident  leading  up  to  the  first  (about  45  hits  per  lesson) 
and  second  GRs  (about  28  hits  per  lesson)  and  the  final  (55  hits  in  the  four  days  before  the 
final)  (Figure  5).  The  period  leading  up  to  the  first  student-led  review  (SLR)  and  GR,  and  to 
the  final,  were  characterized  by  approximately  logarithmic  increases  in  hits  as  a  function  of 
elapsed  time,  while  the  increase  during  the  period  leading  up  to  the  second  SLR  and  GR  was 
less  steep  (Figure  6). 

Table  1.  Intercorrelation  data. 


Lesson  25 

Lesson  42 

Usefulness 

Quality 

Concept 

Usefulness 

Quality 

Concept 

Hits 

0.150 

0.152 

0.075 

0.135 

*0.385 

0.065 

Usefulness 

**0.535 

**0.415 

0.211 

**0.591 

Quality 

**0.425 

*0.340 

(*p  <  0.05;  **p<  0.01) 


The  relationships  (Spearman  r)  among  the  various  ratings  and  frequency  of  access 
were  as  shown  in  Table  1 .  All  of  the  relationships  were  positive.  Several  relationships  were 
statistically  significant,  as  indicated.  Perceptions  of  site  usefulness  and  quality  and  the  web 
server  concept  rating  were  positively  intercorrelated  at  statistically  significant  levels  at  lesson 
25  but  less  strongly  intercorrelated  at  lesson  42.  In  particular,  the  positive  relationship 
between  perceived  quality  and  perceived  usefulness  appeared  to  decline  from  lesson  25  to 
lesson  42.  The  distribution  of  tide  reported  number  of  hits,  which  was  cumulative  across  the 
course,  shifted  from  lesser  to  greater  right  skew  as  more  users  accrued  more  hits,  even  though 
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the  mode  remained  at  1  to  5  hits.  The  positive  relationship  between  the  reported  number  of 
hits  and  the  perception  of  site  quality  appeared  to  increase  from  lesson  25  to  lesson  42. 

The  practical  lesson  was  related  to  the  problem  of  dealing  with  different  file  naming 
and  file  formatting  conventions.  The  web  server  was  fully  Windows  95®  compliant.  Thus,  it 
dealt  adequately  with  file  names  longer  than  8  characters.  The  users'  reported  that  their 
computers  still  had  remnants  of  earlier  operating  systems  that  would  not  accept  file  names 
longer  than  8  characters.  Some  users  used  Word  6.0®  while  others  used  Word  97®.  The 
word  processing  documents  on  the  web  site  were  prepared  using  the  latter,  which  produced 
file  formats  that  were  essentially  unreadable  by  the  former.  The  solution  was  to  post  these 
documents  on  the  web  site  usmg  file  names  of  8  or  fewer  characters  and  to  export  them  from 
Word  97®  in  rich  text  format  (RTF),  which  was  readable  by  Word  6.0®.  Of  course,  the 
export  function  produced  file  name  extensions  of  "RTF"  by  default.  Many  users  did  not  have 
an  association  in  their  Windows  system  between  the  RTF  file  name  extension  and  their  word 
processing  program.  Thus,  they  perceived  an  inability  to  open  the  files.  To  offset  this 
problem,  the  files  on  the  web  site  were  renamed  so  that  they  all  had  DOC  file  name 
extensions,  an  extension  recognized  by  default  by  Word  6.0®  and  Word  97®.  The  free 
Microsoft  translation  that  would  have  allowed  these  Word  6  users  to  read  Word  97  files  was 
not  widely  disseminated  among  the  users  in  this  study. 

Discussion 

The  server  concept  and  specific  site  received  reasonably  good  reviews.  The  modes  of 
the  distributions  of  ratings  for  the  usefulness  of  the  site,  site  information  quality  and  the 
intranet  serve  concept  through  the  first  GR  were  all  "Good"  (a  rating  of  7)  on  a  scale  ranging 
from  "Extremely  poor"  (1)  to  "Excellent"  (9).  All  of  the  distributions  were  somewhat  right- 
skewed,  with  just  a  few  individuals  providing  low  ratings.  These  results  are  supportive  of 
continued  efforts  to  provide  information  to  cadets  in  this  manner. 

The  patterns  of  site  hits  as  functions  of  lesson  occurrence  and  of  calendar  days  were 
predictable.  Not  much  activity  early  on,  with  a  sharp  increase  as  the  first  SLR  and  GR 
approached.  This  was  followed  by  decreased  activity,  a  secondary  increase  as  the  second 
SLR  and  GR  approached,  decreased  activity  again,  and  a  sharp  increase  just  before  the  final. 
The  decision  to  provide  hard  copies  of  the  lesson  plan  notes,  present  on  the  web  site,  to  the 
cadets  probably  contributed  to  the  reduction  in  hit  rate  observed  after  the  first  GR.  The  curve 
provided  a  useful  quantification  of  the  pattern  of  studying  for  the  first  GR,  with  a  time 
constant  (63.2%  of  the  cadets  active)  of  about  12  lessons  for  an  exam  at  lesson  18. 

The  decline  of  the  positive  relationship  between  perceived  quality  and  perceived 
usefulness  from  lesson  25  to  lesson  42  probably  reflected  users'  frustrations  in  gaining  access 
to  the  site.  The  site's  quality  did  not  change  from  lesson  25  to  lesson  42  and  the  distributions 
of  the  users'  perceptions  of  site  quality  were  similar  at  lessons  25  and  42.  However,  the 
distribution  of  ratings  of  perceived  usefulness  was  much  more  varied  at  lesson  42  than  the 
distribution  at  lesson  25.  Apparently,  many  users  indicated  by  their  usefulness  ratings  at 
lesson  42  that  the  quality  of  the  material  at  the  site  was  only  potentially  useful  because  access 
was  unreliable. 

There  was  a  moderate  positive  relationship  between  perceived  usefulness  of  this  site 
and  the  more  general  rating  of  the  server  concept,  at  both  lesson  25  and  lesson  42.  This 
pattern  suggested  that,  for  students  with  limited  exposure  to  web  site  class  management 
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techniques,  the  two  questions  did  not  produce  very  independent  responses.  The  more  general 
question  need  not  be  used  for  relatively  naive  users. 

The  apparent  increase  in  the  positive  relationship  between  the  reported  number  of  hits 
and  the  perception  of  site  quality  from  lesson  25  to  lesson  42  was  probably  caused  by  the 
naturally  occurring  increase  in  tihe  variance  in  the  number  of  hits  accrued  by  users.  This 
effect  should  be  viewed  simply  as  a  statistical  artifact  of  increased  homoscedasticity. 
However,  the  positive  relationship,  itself,  may  have  reflected  higher  numbers  of  hits  by  those 
users  who  perceived  higher  quality  in  the  site's  materials,  and  vice  versa. 

The  unreliability  of  the  LAN  produced  a  tendency  toward  a  misplaced  trust  in  the 
network  as  a  communication  tool,  leading  to  user  frustration  and  unsolicited  comments  on  the 
questionnaires.  Three  of  the  five  unsolicited  comments  provided  by  users  at  lesson  25 
revealed  this  frustration; 

•  "Not  able  to  get  on  for  some  time." 

•  "I  do  not  like  the  web;  it  is  difficult  and  it  never  seems  to  work." 

•  "I  wish  I  had  hard  copies  because  the  Internet  (sic)  is  EXTREMELY  unreliable;  it  is  very 
intermittent  and  you  cannot  depend  on  it  as  a  study  source." 

Six  of  the  ten  imsolicited  comments  provided  by  users  at  lesson  42  revealed  also  the 
frustration: 

•  "The  net  being  down  kept  me  from  accessing  the  site." 

•  "Sometimes  it  [the  net]  works,  sometimes  it  doesn't  (60/40  maybe)  and  that's  too 
unreliable  when  my  grade  depends  on  it." 

•  "I  have  trouble  getting  on  the  intraweb  often.  Not  as  unreliable  as  the  web  though.  The 
Internet  is  useful,  but  I  would  rather  have  hard  copies  because  our  net  is  extremely 
unreliable.  In  fact,  when  I  try  to  use  it,  it  is  down  more  than  up." 

•  "The  server  is  down  too  often." 

•  "  [Usefulness  is  excellent]  when  it  works ! ! ! " 

•  "The  website  was  OK,  but  the  lack  of  Intranet  makes  the  website  hit  or  miss.  It  seems  like 
many  cadets  after  experiencing  2  or  3  failures  don't  even  try  anymore." 

These  anecdotal  representations  of  frustration  were  underscored  by  the  quantitative 
finding  that  the  rating  of  the  server  concept  declined  reliably  from  lesson  25  to  lesson  42. 

Conclusions 

For  an  instructor  versed  in  HTML  or  the  Netscape  Composer®  or  Microsoft 
Frontpage®  editing  environments,  the  personal  web  server  concept  will  prove  to  be  a  useful 
tool  for  the  dissemination  of  information  to  cadets  and  other  students.  This  conclusion  must 
be  caveated  as  follows:  The  users  must  all  have  access  to  the  site  and  be  able  to  use  web 
browser  software,  and  the  network  must  function  reliably. 
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Figure  2.  Ratings  of  site  usefulness. 
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Figure  3.  Ratings  of  site  quality. 
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Figure  5.  Site  hit  rate  as  a  function  of  lesson  number. 
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Figure  6.  Site  hit  rate  as  a  function  of  calendar  date. 
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Appendix  A:  Anonymous  BS  373  Web  Site  Questionnaire 

1 .  How  many  times  have  you  accessed  the  BS  373  site  this  semester  (mark  one)? 
_ Unwilling  or  vmable  to  access  the  site  (go  to  question  5) 

_ 1  to  5  times  _ 6  to  10  times 

_ 11  to  15  times  _ 16  to  20  times 

_ 21  to  25  times  _ 26  to  30  times 

_ more  than  30  times 

2.  Please  rate  the  usefulness  of  the  BS  373  site  for  acquiring  lecture  information. 
Excellent 


_ Good 

_ Borderline 

_ Not  very  good 

_ Extremely  poor 

3.  Please  rate  the  quality  of  the  lecture  information  on  the  BS  373  site. 
Excellent 


Good 

Borderline 


_ Not  very  good 

_ Extremely  poor 

4.  Please  rate  the  usefulness  of  the  Intranet  web  server  concept  for  acquiring  class 
information. 

_ Excellent 

_ Good 

Borderline 


Not  very  good 


_ Extremely  poor 

5.  If  you  are  having  problems  on  your  computer  with  e-mail,  access  to  the  intranet,  access  to 
the  web,  or  other  operability  problems,  and  wish  to  have  help  from  DF  computing,  please 
sign  up  on  the  other  sheet  that  will  be  handed  arovmd. 
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Abstract:  This  investigation,  the  first  in  a  series  of  multi-disciplinary  studies,  will  assess  the  effect  of 
portable  computers  on  the  faculty’s  ability  to  contribute  to  teaching,  research,  and  service.  The 
investigation  employs  a  mixed  design.  The  experimental  group  consists  of  faculty  members  who 
volunteered  to  turn  in  their  desktop  computers  for  notebook  computers.  The  control  group  consists  of 
faculty  members  who  requested  and  received  new,  replacement  desktop  computers  with  a  different 
operating  system  than  they  had  been  using.  A  series  of  six  surveys  were  scheduled  for  administration 
on-line  throughout  one  calendar  year  to  collect  longitudinal  data  on  hardware  and  software  use,  ease  of 
use,  and  frequency  of  use  within  the  experimental  and  control  groups.  We  hypothesized  here  that  the 
two  groups  would  hot  differ  with  respect  to  age,  military-civilian  mix,  gender,  academic  rank,  military 
rank,  computer  experience,  teaching  experience,  and  length  of  time  at  USAF  A.  Most  hypotheses  were 
supported.  However 

The  US  Air  Force  Academy  had  considered  the  option  of  issuing  portable  notebook 
computers  to  cadets  in  place  of  desktop  computers  for  several  years.  The  US  Military 
Academy,  West  Point  NY,  had  conducted  a  small  pilot  study  of  notebook  computers  in  1994. 
Eighteen  cadets  (one  classroom  group)  were  given  notebook  computers  for  the  entire  spring 
semester.  The  notebook  computer  was  chosen  to  match  the  capabilities  of  the  cadet  desktop 
model.  Due  to  the  high  failure  rate  and  high  cost  of  the  notebook  computers  (when  compared 
to  desktops),  the  researchers  concluded  that  notebook  computers  were  not  yet  acceptable 
replacements  for  desktops. 

Mayville  State  University  (MSU)  and  Valley  City  State  University  (VCSU)  in  North 
Dakota  had  adopted  notebook  computers  for  all  students  and  faculty  in  1996.  They  reported 
overwhelmingly  favorable  results  including  increased  level  of  communication  on  campus, 
improved  student  retention  rates,  increased  instructor  use  of  technology,  improved  research 
abilities,  and  improved  ability  for  students  to  work  in  groups.  When  considering  these  results, 
please  we  noted  that  the  MSCWCSU  study  confounded  the  effects  of  issuing  computers  to  all 
students  with  the  effects  of  notebook  computers,  themselves. 

Most  researchers  had  agreed  on  the  advantages  of  notebook  computers,  citing  the 
opportunity  to  bring  computers  to  class,  the  ability  to  bring  computers  on  athletic  and  other 
trips,  ease  of  storing  computers,  ease  of  transporting  a  failed  computer  for  repair,  equal  access 
to  computers  across  disciplines,  increased  number  of  content  delivery  options,  and  ^e  ability 
to  create  a  lab  environment  in  any  classroom. 

Researchers  had  also  tended  to  agree  on  the  disadvantages  of  notebook  computers. 

The  West  Point  researchers  were  disturbed  by  the  failure  rate  among  notebook  computers 
(31%  of  the  computers  failed  during  the  study).  Other  disadvantages  included  the  12  to  18 


78 


month  “technology  gap”  for  notebooks,  lack  of  upgrade  capability,  short  battery  life, 
increased  repair  costs,  and  ergonomics  problems  associated  with  the  small  screen  and  small 
keyboard. 

Since  the  West  Point  study,  notebook  computers  had  become  less  expensive,  more 
powerful  and  more  durable.  A  listing  of  “Notebook  Colleges  and  Universities”  included  over 
50  institutions^.  The  time  was  ripe  for  USAFA  to  conduct  its  own  investigations  to  assess  the 
feasibility  of  notebook  computers  as  replacements  for  cadet  desktop  computers.  Fiscal  Year 
98  fallout  money  provided  funding  for  100  Pentium  II  notebook  computers  for  faculty. 

This  investigation  was  planned  as  the  first  in  a  series  of  multi-disciplinary  studies 
designed  to  assess  alternatives  to  traditional  desktop  computers.  Specifically,  the  plan  was  to 
assess  the  effect  of  portable  computers  on  the  faculty’s  ability  to  contribute  to  teaching, 
research,  and  service.  The  objectives  were  to  determine: 

•  The  criteria  under  which  notebook  computers  were  feasible  alternatives  to  desktop 
computers. 

•  How  notebook  computers  affected  the  faculty’s  teaching,  research,  and  service. 

•  Which  features  of  notebook  and  desktop  computers  faculty  used. 

•  Which  software  products  faculty  used  on  notebook  and  desktop  computers. 

•  The  criteria  under  which  a  similar  investigation  should  be  conducted  with  cadets. 

•  A  statistical  basis  to  facilitate  the  identification  of  potential  applications  of  notebook 
computers  for  the  Air  Force  and  Department  of  Defense. 

This  paper  reports  the  results  of  a  descriptive  and  comparative  analysis  of  entry  survey 
data,  collected  at  the  beginning  of  the  faculty  project,  including  user  demographics  and  prior 
experience  with  computer  use. 

Methods 

Experimental  Design.  The  investigation  used  a  mixed  design.  The  experimental 
group  consisted  of  faculty  members  who  volunteered  to  turn  in  their  desktop  computers  for 
notebook  computers.  Both  the  older  desktop  and  newer  notebook  computers  used  the 
Windows98  operating  system.  The  control  group  consisted  of  faculty  members  who 
requested  and  received  new,  replacement  Pentium  II  desktop  computers  with  a  different 
operating  system  than  they  had  been  using  (Windows  NT  replaced  Windows  95/98).  A  series 
of  six  surveys  were  scheduled  for  administration  on-line  throughout  one  calendar  year 
(November  1998  -  December  1999)  to  collect  longitudinal  data  on  hardware  and  software 
use,  ease  of  use,  and  frequency  of  use  within  the  experimental  and  control  groups.  Other 
sources  of  data,  not  reported  here,  included  maintenance  logs,  technology  training  rosters, 
network  logs,  and  anecdotal  reports.  Descriptive  statistics  were  used  to  summarize  the  group 
data  and  parametric  and  non-parametric  metiiods  were  used  to  compare  the  two  groups.  We 
hypothesized  that  the  two  groups  would  not  differ  with  respect  to  age,  military-civilian  mix, 
gender,  academic  rank,  military  rank,  and  computer  experience. 

Subjects.  Members  of  the  experimental  group  were  a  subset  of  over  100  volunteers 
who  gave  up  their  desktop  computers  in  exchange  for  notebook  computers.  Since  there  were 
more  volunteers  than  computers,  the  investigators  selected  those  who  were  teaching  and  who 


^  http://www.vcsu.nodak.edu/offices/itc/notebooks/other.htm 
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had  more  than  one  year  left  at  USAFA.  The  control  group  was  a  subset  of  150  faculty 
members  who  received  new  desktop  computers.'® 

Instruments.  The  survey  package  contained  three  surveys:  Entry,  Periodic  and  Exit.  The 
subjective  rating  scales  in  the  surveys  were  constructed  with  reference  to  Babbitt  and  Nystrom 
(1989)  to  help  assure  that  respondents  would  differentiate  among  scale  anchors.  Only  data 
from  the  Entry  Survey  are  reported  here. 

Procedures.  The  Entry  Survey  was  taken  by  experimental  group  participants  shortly 
before  their  notebook  computers  were  issued,  and  by  control  group  participants  shortly  after 
their  new  desktop  computers  were  issued.  The  Periodic  Survey  was  to  be  given  six  times, 
with  a  specification  each  time  about  the  computer  and  the  period  that  was  to  be  rated: 

1 .  In  conjunction  with  the  Entry  Survey.  The  survey  applied  to  the  participant's  experience 
with  his  or  her  older  USAFA  desktop  computer  during  the  preceding  three  monAs 

2.  Mid-spring  semester  1999.  The  survey  was  to  apply  to  the  participant's  experience  with 
his  or  her  new  USAFA  computer  during  the  first  half  of  the  spring  semester. 

3.  End  of  the  spring  semester  1999.  The  survey  was  to  apply  to  the  participant's  experience 
with  his  or  her  new  USAFA  computer  during  the  second  half  of  the  spring  semester. 

4.  End  of  the  summer  1999.  The  survey  was  to  apply  to  the  participant's  experience  with  his 
or  her  new  USAFA  computer  during  the  summer. 

5.  Mid-fall  semester  1999.  The  survey  was  to  apply  to  the  participant's  experience  with  his 
or  her  new  USAFA  computer  during  the  first  half  of  the  fall  semester. 

6.  End  of  the  fall  semester  1999.  The  survey  was  to  apply  to  the  participant's  experience 
with  his  or  her  new  USAFA  computer  during  the  second  half  of  the  fall  semester. 

The  Exit  Survey  was  to  be  given  in  conjunction  with  the  6*  Periodic  Survey.  The  Entry 
Survey,  from  which  the  data  reported  here  were  extracted,  is  shown  as  an  attachment.  The 
surveys  were  implemented  in  hypertext  markup  language  (HTML)  and  common  gateway 
interface  (CGI)  on  a  personal  web  server  coimeeted  to  the  USAFA  Intranet.  The  data  were 
stored  automatically  in  comma-delimited  ASCII  text  files  and  imported  into  spreadsheet 
software  for  data  reduction. 

Group  differences  in  demographics  were  assessed,  for  parametric  data,  by  T  test,  with 
homogeneity  of  variance  assessed  by  Levene  F,  and  differences  were  assessed  by  for  raw 
(not  proportional),  non-parametric,  demographic  data.  The  users  also  provided  six  subjective 
ratings  on  a  scale  of  1  (very  unsatisfactory)  to  6  (very  satisfactory),  with  only  those  two 
anchor  phrases  on  the  scale.  The  six  dimensions  rated  were  labeled  "personal  computers," 
"laptop/notebook  computers,"  "Internet  e-mail,"  "USAFA  Intranet,"  "web  browsing"  and  "the 
USAFA  dial-in  service."  Evidence  of  differences  between  groups  was  sought  using  the 
Kolmogorov-Smimov  2-sample  test  (Siegel,  1956). 

Results 

As  of  8  January  1999,  86  Notebook  users  and  48  Desktop  users  had  responded  to  the 
Entry  Survey.  The  age  distributions  of  the  two  groups  were  not  significantly  different.  The 
notebook  user  mean  age  was  36.4  ±  7.0  yr  and  the  desktop  mean  was  37.5  ±  8.6  yr. 

The  military-civilian  mix  and  the  distributions  of  military  ranks  were  nearly  identical 
across  groups,  as  shown  in  Figure  1.  The  mode  of  this  distribution  was  03  (Captain),  with  a 


The  project  was  submitted  for  review  by  the  USAFA  Institutional  Review  Board,  but  was  judged  by  the  IRB 
to  be  a  "normal  educational  practice"  and  thus  was  not  reviewed  by  the  IRB. 
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secondary  mode  for  civilians.  Proportions  of  male  and  female  users  were  similar  across 
groups:  19.8%  of  Notebook  users  and  18.8%  of  Desktop  users  were  female.  The 
distributions  of  academic  ranks  were  nearly  identical  across  groups  as  shown  in  Figure  2,  with 
the  mode  falling  at  Assistant  Professor.  The  histogram  of  the  distribution  of  the  two  kinds  of 
computers  across  academic  divisions  (Figure  3)  revealed  an  apparent  tendency  for 
Engineering  faculty  to  select  the  desktop  configuration  over  the  notebook  configuration  (X^  = 
5.16,  df  =  3,  p  <  0.20).  The  distributions  of  academic  degrees  were  nearly  identical  across 
groups  as  shown  in  Figure  4,  with  the  mode  falling,  interestingly,  at  the  doctoral  level. 

The  total  amount  of  computer  experience  did  not  differ  significantly  between  the  two 
groups.  The  Notebook  group  reported  15.5  ±  5.7  yr  and  the  Desktop  group  reported  16.6  ±  4.8 
yr.  Similarly,  the  total  amount  of  personal  computer  experience  did  not  differ  significantly 
between  the  two  groups.  The  Notebook  group  reported  12.6  ±  3.7  yr  and  the  Desktop  group 
reported  13.4  ±  4.8  yr.  The  two  groups  also  did  not  appear  to  differ  significantly  with  respect 
to  previous  notebook  computer  experience,  though  the  distributions  were  significantly  right- 
skewed.  Both  groups  reported  an  average  of  about  2  to  3  yr  of  previous  notebook  computer 
experience. 

Experience  using  e-mail  did  not  differ  significantly  between  groups.  The  Notebook 
group  reported  5.8  ±  3.2  yr  and  the  Desktop  group  reported  6.3  ±  2.2  yr  of  e-mail  use. 
Similarly,  experience  with  the  USAFA  Intranet  did  not  differ  significantly  between  groups. 
The  Notebook  group  reported  2.4  ±  2.4  yr  and  the  Desktop  group  reported  2.6  ±  2.6  yr  of 
Intranet  experience.  Experience  using  the  World  Wide  Web  (WWW)  did  not  differ 
significantly  between  groups.  The  Notebook  group  reported  3.6  ±  1.7  yr  and  the  Desktop 
group  reported  3.8  ±  1 .5  yr  of  WWW  use. 

The  two  groups  also  did  not  appear  to  differ  significantly  with  respect  to  experience 
Avith  the  USAFA  dial-in  service,  though  the  distributions  were  significantly  right-skewed. 
Both  groups  reported  an  average  of  about  0.5  yr  of  previous  notebook  computer  experience. 
Experience  with  other  providers'  dial-in  services  did  not  differ  significantly  between  groups. 
The  Notebook  group  reported  2.9  ±  3.1  yr  and  the  Desktop  group  reported  2.1  +  2.1  yr  of 
other  dial-in  use. 
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Figure  1.  Military-civilian  mix  for  the  Notebook  and  Desktop  groups. 
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Figure  2.  Academic  rank  for  the  Notebook  and  Desktop  groups. 
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Academic  Division 
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Figure  3.  Distributions  of  computers  across  academic  divisions. 


Academic  Degree 


60.0% 


50.0% 


40.0% 


30.0%  ^ 


20.0% 


10.0% 


0.0% 


Other 


Bachelors 


Masters 


Doctorate 


Figure  4.  Academic  degrees  for  the  Notebook  and  Desktop  users. 

The  six  subjective  ratings  were  provided  on  a  scale  of  1  (very  imsatisfactory)  to  6 
(very  satisfactory),  with  only  those  two  anchor  phrases  on  the  scale  (Figure  5).  Again,  the  six 
dimensions  were  labeled  "personal  computers,"  "laptop/notebook  computers,"  "Internet  e- 
mail,"  "USAFA  Intranet,"  "web  browsing"  and  "the  USAFA  dial-in  service." 

The  groups  did  not  differ  significantly  with  respect  to  their  ratings  of  personal 
computers.  The  mode  of  the  distribution  was  a  rating  of  5,  with  a  range  from  1  to  6.  The 
groups  did  not  differ  significantly  with  respect  to  their  ratings  of  laptop/notebook  computers. 
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The  mode  of  the  distribution  was  a  rating  of  4.5,  with  a  range  from  1  to  6.  There  were  62 
users  in  the  notebook  group  and  13  users  in  the  desktop  group  who  provided  ratings. 

The  groups  did  not  differ  significantly  with  respect  to  their  ratings  of  e-mail.  The 
mode  of  the  distribution  was  a  rating  of  5,  with  a  range  from  1  to  6.  There  were  80  users  in 
the  notebook  group  and  23  users  in  the  desktop  group  who  provided  ratings. 

The  groups  differed  significantly  with  respect  to  their  ratings  of  the  USAFA  Intranet 
(X^  =  16.3,  df  =  2,  p  <  0.002).  The  mode  of  the  notebook  group  distribution  was  a  rating  of  5, 
with  a  range  from  1  to  6.  The  mode  of  the  desktop  group  distribution  was  a  rating  of  2,  with  a 
range  from  1  to  5.  There  were  83  users  in  the  notebook  group  and  22  users  in  the  desktop 
group  who  provided  ratings. 

The  groups  did  not  differ  significantly  with  respect  to  their  ratings  of  web  browsing. 
The  mode  of  the  distribution  was  a  rating  of  5,  with  a  range  from  1  to  6.  There  were  84  users 
in  the  notebook  group  and  23  users  in  the  desktop  group  who  provided  ratings. 

The  groups  did  not  differ  significantly  with  respect  to  fiieir  ratings  of  the  USAFA  dial- 
in  service.  The  mode  of  the  distribution  was  a  rating  of  1,  with  a  range  from  1  to  6.  There 
were  37  users  in  the  notebook  group  and  9  users  in  the  desktop  group  who  provided  ratings. 


Ratings 


Figure  5.  Subjective  ratings  of  satisfactoriness  (6  is  high)  for  personal  computers, 
notebook  computers,  e-mail,  the  USAFA  Intranet,  the  World  Wide  Web,  and  the  USAFA 
dial-in  service.  Figures  in  parentheses  are  sample  size.  The  ratings  for  the  Intranet  differed 

between  groups  (see  text). 


Discussion 

We  hypothesized  that  the  experimental  and  control  groups  would  not  differ  with 
respect  to  age,  military-civilian  mix,  gender,  academic  rank,  military  rank,  and  computer 
experience.  These  hypotheses  were  supported.  The  pooled  mean  age  was  about  37  yr  and  the 
pooled  male-female  ratio  was  about  81%  to  19%.  The  pooled  mode  for  academic  rank  was 
Assistant  Professor  and  for  military  rank  was  03  (Captain),  with  a  secondary  mode  for 
civilian  faculty. 
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The  pooled  mean  for  all  computer  experience  was  about  16  yr,  for  personal  computer 
experience  about  13  yr,  and  for  notebook  computer  experience  about  2.75  yr.  The  pooled 
mean  for  e-mail  experience  was  about  6  yr,  for  Intranet  experience  about  2.5  yr,  for  Web 
experience  about  3.7  yr,  for  USAFA  dial-in  experience  about  0.75  yr,  and  for  other  dial-in 
experience  about  2.5  yr. 

While  the  mode  of  military  rank  and  civilian-military  mix  (Figure  1)  occurred  at  the 
level  of  03  (Captain),  the  mode  of  the  distribution  of  education  level  occurred  at  the  doctoral 
level  (Figure  4).  This  was  somewhat  interesting  because  most  Captains  on  the  faculty  were 
known  to  be  at  the  masters  level.  The  secondary  mode  for  civilian  faculty  (Figure  1)  probably 
caused  the  education  level  mode  to  occur  at  the  doctoral  level.  All  civilian  faculty  are 
required  to  hold  a  doctoral  level  degree. 

Generally,  the  two  groups  came  into  the  project  satisfied  with  various  aspects  of 
computer  use.  They  rated  the  satisfactoriness  of  personal  computers,  notebook  computers,  e- 
mail  and  the  Web  rather  highly  (4.5  to  5  on  a  scale  of  1  to  6).  However,  they  rated  the 
USAFA  dial-in  service  as  low  as  possible  on  the  scale.  This  low  rating  may  signal  a  problem 
that  should  be  dealt  with,  especially  for  the  Notebook  group,  as  the  project  continues. 

The  two  groups  differed  significantly  Avith  respect  to  their  ratings  of  the 
satisfactoriness  of  the  USAFA  Intranet.  The  mode  of  the  notebook  group  distribution  was 
high  (satisfactoriness  =  5),  while  the  mode  of  the  desktop  group  distribution  was  low 
(satisfactoriness  =  2).  The  reason  for  this  difference  was  obscure.  One  may  speculate  that  our 
users  who  acquired  notebook  computers  were  more  likely  to  have  had  experience  with  other 
intranets  and,  thus,  used  a  larger  context  for  this  particular  rating.  However,  there  were  no 
data  available  to  support  or  refute  this  speculation. 

Two  sets  of  data  were  missing  from  the  Entry  Survey:  years  teaching  experience,  and 
length  of  time  at  USAFA.  These  items  will  be  covered  on  the  Exit  Survey. 

The  data  acquired  to  date  and  to  be  acquired  in  this  project  should  provide  a  statistical 
basis  that  may  help  identify  potential  applications  of  notebook  computers  across  many  Air 
Force  and  Department  of  Defense  missions. 
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Integrating  Mathematica  with  PowerPoint 

Frank  C.  Wilson,  M.S.,  Captain,  USAF 
Department  of  Mathematical  Sciences 

Abstract:  Mathematica  is  a  powerful  graphics  generation  tool,  and  PowerPoint  is  a  powerful 
instructional  delivery  tool.  By  marrying  the  two  products,  difficult  mathematical  concepts  may  be 
presented  graphically  in  a  more  understandable  way.  A  sampling  of  PowerPoint  slides  using 
Mathematica  graphics  were  presented  along  with  an  explanation  of  the  graphics  manipulation  process. 

Several  examples  of  PowerPoint  presentations  developed  for  the  Math  141  calculus 
cotirse  were  demonstrated  in  a  10-15  minute  presentation.  Thirty-one  PowerPoint 
presentations  have  been  developed  to  support  the  Math  141  calculus  course.  In  this 
presentation,  the  process  of  converting  Mathematica  graphs  into  PowerPoint  objects  was 
discussed.  The  method  is  detailed  in  the  steps  below. 

Converting  Mathematica  Graphs  to  PowerPoint  Objects 

Copy  the  graphic  in  Mathematica 

Click  on  the  graphic  and  press  “Ctrl-C” 

Paste  the  graphic  in  PowerPoint 

Press  “Ctrl-v”  to  paste  the  graphic  in  an  open  slide. 

Size  the  graphic 

Click  on  the  handles  and  drag  to  the  desired  dimension. 

Convert  the  graphic  to  a  Microsoft  Office  drawing 

Double-click  the  image  and  select  “Yes”  (This  makes  each  element  of  the  graph  a 
modifiable  object.) 

If  the  font/line  color  of  the  graph  is  similar  in  color  to  the  slide  background,  change  the 
font  and  line  color  to  a  contrasting  color 

On  the  drawing  toolbar,  click  the  arrow  next  to  paintbrush  icon  and  select  a  color. 

(This  changes  the  line  color.) 

On  the  drawing  toolbar,  click  the  arrow  next  to  “A”  icon  and  select  a  color.  (This 
changes  the  line  color.) 

Remove  the  white  bacl^ound  of  the  graph. 

Click  on  the  slide  background.  (This  deselects  all  selected  objects.) 

Click  on  the  white  space  of  the  graph 
Press  the  Delete  Key 

Repeat  the  above  delete  sequence  two  more  times. 
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Group  the  x-axis  labels 

Hold  the  Shift  Key  down  and  mouse  click  on  each  label 

Click  Draw  on  the  drawing  toolbar  and  select  Group.  (This  groups  the  labels) 

Group  the  y-axis  labels 

Hold  the  Shift  Key  down  and  mouse  click  on  each  label 

Click  Draw  on  the  drawing  toolbar  and  select  Group.  (This  groups  the  labels) 

Remove  the  line  border  from  the  x-  and  y-axis  labels 

Hold  the  Shift  Key  down  click  on  an  x-axis  label  and  a  y-axis  label 

On  the  drawing  toolbar,  click  the  arrow  next  to  paintbrush  icon  and  scroll  to  No  Line. 
(This  makes  the  line  invisible.) 

If  necessary,  adjust  the  position  of  the  x-  and  y-axis  labels 
Click  on  the  appropriate  label  group 
Use  the  arrow  keys  to  adjust  the  label  position 

Increase  the  width  of  the  x-  and  y-  axis 

Hold  the  Shift  Key  down  click  on  the  x-axis  and  the  y-axis 

On  the  drawing  toolbar,  click  on  the  line  thickness  icon  (three  lines  of  different  widths 
are  on  the  face  of  the  icon). 

Scroll  to  1  Vz  pt  (This  changes  the  line  width  to  1  Vz  pt) 

Change  the  line  width/color  of  the  plot  of  the  function 
Click  on  the  plot  of  the  function 

On  the  drawing  toolbar,  click  on  the  line  thickness  icon  (three  lines  of  different  widths 
are  on  the  face  of  the  icon). 

Scroll  to  3  pt  (This  changes  the  line  width  to  3  pt) 

On  the  drawing  toolbar,  click  the  arrow  next  to  paintbrush  icon  and  scroll  to  the 
desired  line  color.  (This  makes  the  line  invisible.) 

If  you  have  multiple  plots,  repeat  the  above  steps  for  each  plot. 

If  desired,  label  plots  with  PowerPoint  text  box. 

Animating  PowerPoint  Objects 

Right-click  on  the  object  you  want  to  animate. 

Scroll  to  Custom  Animation 

In  the  Custom  Animation  dialog  box,  click  on  the  arrow  next  to  No  Effect. 

Scroll  down  to  the  desired  animation  effect.  (Wipe  Right  works  well.) 

Click  the  Preview  button  to  see  the  effect. 

Click  the  OK  button  when  you’re  satisfied  with  the  animation  effect. 
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ASSESSMENT 


Developing  a  Role  for  Students  in  Higher  Education  Outcomes  Assessment:  Self  Study 

Committee  Participation  (Panel) 

Anthony  J.  Aretz,  Ph.D.,  Lt  Colonel,  USAF 
Department  of  Behavioral  Sciences  and  Leadership 

Panel  members:  C2C  Lindsay  L.  Winter,  C2C  Andrew  J.  Nutz,  C2C  Laura  C.  Terry,  and 

C2C  Reed  W.  Wangerud 

Department  of  Behavioral  Sciences  and  Leadership 

Abstract:  A  unique  aspect  of  the  NCA  Self  Study  Subcommittee,  which  was  tasked  with  determining 
whether  or  not  the  Academy  was  accomplishing  its  educational  and  other  purposes,  was  its  inclusion 
of  four  second  class  cadets  as  full  fledged  members  of  the  committee  for  the  fall  semester  of  1998.  The 
cadets'  contributions  to  the  committee  process  and  report  were  both  positive  and  substantial.  The  role 
of  this  participation  in  the  cadets'  development  also  appears  to  have  been  very  beneficial.  Each  cadet 
panel  member  described  what  they  did  and  told  what  they  learned  about  themselves,  their  institution 
and  their  own  education. 

A  unique  aspect  of  the  NCA  Self  Study  Subcommittee  tasked  with  determining 
whether  or  not  the  Academy  was  accomplishing  its  educational  and  other  purposes  was  its 
inclusion  of  four  second  class  cadets  as  full  fledged  members  of  the  committee  for  the  fall 
semester  of  1998.  The  cadets'  contributions  to  the  committee  process  and  report  were  both 
positive  and  substantial.  The  role  of  this  participation  in  the  cadets'  development  also  appears 
to  have  been  very  beneficial. 

Cadets,  nominated  by  existing  subcommittee  members,  were  given  the  opportunity  to 
eimoll  in  Behavioral  Sciences  495,  Educational  Outcomes  Assessment  Practicum,  during  the 
fall  semester  of  1998.  The  course  was  1.5  semester  hours  and  DFR  assisted  in  making 
scheduling  arrangements  so  cadets  could  attend  weekly  committee  meetings.  Initial  meetings 
were  devoted  to  defining  the  committee's  task  and  organizing  our  resources  to  accomplish  it 
within  necessary  time  constraints.  When  it  became  clear  that  further  progress  would  require 
the  25-person  subcommittee  would  need  to  divide  itself  into  4  to  5  person  teams  to  look  at 
particular  sets  of  data  or  reports,  the  cadets  each  became  members  of  separate  subcommittees. 
When  each  subcommittee  had  completed  its  tasks,  the  group  met  again  as  a  whole.  Each 
group  presented  its  findings;  other  members  of  the  subcommittee  questioned,  criticized  and 
critiqued  the  presentations.  Throughout  this  process,  cadets'  perspectives  and  insights  helped 
provide  a  reality  check  for  faculty  and  staff  members'  conjectures  and  interpretations. 

The  cadets'  progress  report  was  based  upon  brief  written  reports  describing  what  they 
had  done  thus  far,  how  it  related  to  the  NCA  question  of  whether  or  not  the  institution  was 
accomplishing  its  educational  and  other  purposes,  and  what  they  had  learned  thus  far.  Copies 
of  these  reports  were  distributed  to  about  a  dozen  other  members  of  the  committee  for  their 
review  and  critique.  Cadets  received  these  inputs  and  used  them  to  prepare  their  final  reports 
on  their  participation  on  the  committee.  These  reports  were  included  as  an  attachment  to  the 
NCA  Self  Study. 

Each  cadet  panel  member  briefly  introduced  her/himself,  described  what  they  did,  and 
told  what  they  learned  about  themselves,  their  institution  and  their  own  education. 
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C2C  Lindsay  Winter 

I  participated  in  Lt.  Colonel  Anne  Campbell’s  small  group.  We'were  assigned  to  look 
over  the  1998  Graduation  Survey  and  draw  conclusions  from  the  data.  I  was  specifically 
assigned  to  the  questions  that  involved  the  Athletics  Department  (AH).  I  looked  at  each 
question’s  responses  and  tried  to  find  trends  in  die  data.  I  broke  my  results  down  into  four 
categories:  conunitment  to  physical  fitness,  leadership  skills  and  development,  moral 
development,  and  pride  in  athletic  accomplishments.  I  found  negative  results  in  all  four  areas 
of  AH.  As  a  result,  I  tried  to  come  up  with  some  ideas  to  help  improve  these  areas  of 
athletics. 

During  this  process  I  learned  several  interesting  things  about  myself  I  realized  how 
easily  I  can  lose  my  assertiveness  and  become  intimidated  when  surrounded  by  high  ranking 
officers.  I  also  spent  lots  of  time  searching  for  solutions  to  some  of  the  problem  areas  I 
discovered  in  my  research.  I  realized  how  hard  it  is  to  find  solutions  to  many  of  the  issues  I 
encountered.  I  also  realized  how  cadet  attitudes  are  a  large  portion  of  the  problems  that  I  was 
attempting  to  solve.  I  also  learned  several  interesting  things  about  my  institution  and 
education  through  this  class. 

After  all  my  years  in  school,  I  never  realized  how  much  time  went  into  assessment  and 
on  top  of  that  the  process  of  becoming  an  accredited  university.  I  thought  it  was  great  to  see 
how  involved  the  faculty  was,  and  disheartening  to  see  how  many  refused  to  help  us  achieve 
our  goal  by  providing  us  feedback.  The  opportunity  to  be  a  part  of  this  small  group  was  very 
valuable  to  me.  It  gave  me  an  appreciation  for  the  hard  work  that  teachers  have  to  do  outside 
of  the  classroom.  This  is  an  area  that  greatly  interests  me  because  I  hope  to  come  back  to  the 
Academy  and  teach  someday. 

C2C  Andrew  Nutz 

During  the  conference  on  assessment  in  education,  I  participated  on  a  student  panel 
that  provided  my  fellow  cadets  and  I  the  opportunity  to  express  our  points  of  view  on  the 
Academy’s  self-study  for  the  North  Central  Association’s  (Chapter  5,  Criterion  3).  During 
the  student  panel,  I  provided  details  of  my  participation  with  the  committee  as  well  as  with 
my  sub-committee  assigned  to  examine  miscellaneous  archival  objective  data.  I  gave  my 
opinion  on  the  importance  of  the  self-study  and  assessment  processes  and  I  fielded  a  handful 
of  questions  from  the  guests,  mainly  directed  toward  my  experience. 

A  major  point  that  was  debated  at  the  conference  was  the  fact  that,  although  the 
Academy  uses  numerous  evaluation  tools  to  gather  significant  data  from  a  variety  of  sources 
using  many  different  methods,  the  Academy  does  not  yet  have  a  comprehensive  and 
systematic  process  for  disseminating  data  and  analyses  back  to  decision-makers.  In  addition, 
it  was  evident  during  the  conference  that  the  Air  Force  Academy  places  great  importance  on 
evaluating  mission  accomplishment  and  expends  a  great  deal  of  effort  gathering  data. 

One  of  the  main  points  that  I  addressed  was  that  it  is  not  “looking  good”  for  an 
outsider  that  matters,  but  the  opportunities  and  recommendations  for  improvement  that  can  be 
made.  In  addition,  I  found  that  the  accreditation  process  allows  the  Air  Force  Academy  and 
other  institutions  of  higher  learning  the  opportunity  to  determine  whether  or  not  they  are 
accomplishing  their  educational  and  other  outcomes.  Finally,  I  believe  that  accreditation  is  in 
place  for  institutions  to  evaluate  themselves  and  to  gather  feedback  as  to  how  they  can 
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continuously  improve.  My  overall  experience  was  a  good  one,  during  which  I  learned  a  great 
deal  about  the  process  of  assessment,  as  well  as  about  myself. 

C2C  Laura  Terry 

By  participating  on  the  NCA  assessment  committee,  I  learned  a  great  deal,  not  limited 
to  only  the  realm  of  assessment.  This  brief  exposure  opened  my  eyes  to  the  limited 
perspective  I  had  concerning  what  goes  on  at  the  school. 

I  worked  on  the  sub-committee  that  looked  at  which  relevant  data  could  be  obtained 
from  surveys  returned  by  graduates  and  their  supervisors.  I  was  especially  interested  in 
several  responses  by  graduates  that  showed  concern  as  to  whether  the  survey  would  ever  be 
used  to  change  anything.  This  reflects  a  common  attitude  fearing  the  seeming  lack  of  action 
taken  based  on  cadets’  responses.  Our  group  found  that  in  some  cases,  we  need  to  make  sure 
that  the  assessment  process  is  completed  by  ensuring  that  decision  makers  understand  and  use 
the  data  collected. 

As  cadets,  our  presence  on  the  committee  helped  to  remind  everyone  of  the  main  focus 
of  this  institution,  namely  the  development  of  officer  candidates.  I  cannot  claim  that  this 
particular  group  was  guilty  of  ignoring  this  fact,  but  I  am  certain  that  the  idea  would  be  useful 
for  other  groups  around  the  Academy. 

I  was  given  an  unusual  opporhinity  in  working  with  this  committee.  As  students  we 
rarely  get  to  observe  our  instructors  working  in  a  group  together,  let  alone  being  a  part  of  the 
group.  There  are  many  people  here  who  care  about  providing  the  best  education  possible  and 
seek  to  make  the  necessary  changes.  Many  good  programs  are  in  place  for  collecting  data.  In 
some  cases,  we  need  to  make  sure  that  the  assessment  process  is  completed  by  ensuring  that 
decision  makers  understand  and  use  the  data  collected.  To  be  aware  of  this  perspective,  while 
actually  submersed  in  the  system,  is  a  uniquely  valuable  experience. 
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United  States  Air  Force  Academy  Accreditation  Self-Study  Subcommittee  on  Assessment 
“Is  The  Air  Force  Academy  Accomplishing  It’s  Mission?”  (Panel) 

Larry  A.  Smith,  M.A,  Colonel,  USAF 
34*  Education  Group 

Panel  Members:  Major  John  Putnam  (Dept  of  Biology),  Lt  Col  Anne  Campbell  (Dept  of 
Political  Science),  Major  Julie  Chesley  (Dept  of  Management) 

Abstract:  The  panel  addressed  the  question  of  whether  or  not  the  Air  Force  Academy  is 
accomplishing  its  educational  and  other  purposes.  The  discussion  was  based  on  the  work  of  a 
subcommittee  of  the  North  Central  Association  Self  Study  Committee.  This  committee  worked 
together  for  nearly  a  year  and  collected  a  wide  array  of  data  and  reports  concerning  the  Academy's 
educational  efficacy. 

Background 

The  Air  Force  Academy  is  undergoing  reaccreditation  by  the  North  Central 
Association  (NCA).  A  significant  part  of  that  process  is  preparing  a  self-study  that  details 
how  well  the  institution  is  accomplishing  its  stated  objectives.  One  chapter  of  the  study 
specifically  examines  the  assessment  process  used  to  determine  whether  or  not  the  Academy 
mission  is  being  accomplished.  The  Assessment  Subcommittee  was  tasked  to  write  that 
chapter.  The  panel,  selected  members  from  the  Assessment  Subcommittee,  discussed  the 
methodology  involved  and  the  general  findings  that  resulted. 

Subcommittee  Composition 

The  committee  was  co-chaired  by  Colonel  David  Porter  from  the  academic  arena  and 
Colonel  Larry  Smith  from  the  military  mission  area.  Experts  from  every  Academy  mission 
area  as  well  as  cadets  from  the  student  body  were  represented  on  the  committee. 

Committee  Process 

The  committee  reviewed  existing  data  collection  instruments,  examined  collected  data, 
performed  gap  analysis  and  conducted  some  independent  research  in  the  areas  where  gaps 
existed.  A  report  detailing  this  data  was  compiled  and  sent  to  various  decision  makers  across 
the  Academy  for  comment.  The  committee  then  evaluated  the  data  including  decision  maker 
comments  and  drafted  their  findings  into  the  self-study  assessment  chapter.  The  chapter  was 
submitted  to  the  Academy  Accreditation  Committee  which  is  incorporating  it  into  the  final 
self-study. 

The  assessment  committee  enlisted  cadets  to  help  evaluated  the  Academy’s  mission 
effectiveness.  Top  cadets  from  various  majors  volunteered  to  help  with  the  committee 
process.  They  served  on  various  sub  groups  and  wrote  a  paper  on  their  experiences  with  the 
process.  In  addition,  they  made  a  presentation  to  the  PPIER  Symposium  outlining  what  they 
had  learned.  To  a  person,  the  cadets  significantly  valued  their  chance  to  participate  in  the 
accreditation  process.  For  their  effort,  the  cadets  received  academic  credit  for  an  independent 
study  course. 
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General  Findings 

The  Academy  uses  many  varied  instruments  to  collect  a  great  deal  of  data,  and 
Academy  leaders  use  the  data  to  make  decisions.  Furthermore,  since  the  Academy’s 
graduates  initially  work  for  the  same  employer  unique  possibilities  exist  for  follow  up  data 
from  the  graduate  and  graduate  supervisor/peer  surveys  the  institution  accomplishes.  This 
data  tells  Academy  faculty  and  stadfhow  well  they  meet  the  needs  of  Air  Force  leadership. 
However,  the  committee  also  found  no  central  feedback  process  existed  to  help  make  the 
system  more  responsive. 
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Assessment  in  Higher  Education:  Is  It  Making  a  Difference?  (Panel) 

Randall  J.  Stiles,  Ph.D.,  Colonel,  USAF  • 

Director  of  Education 

Panel  members:  Dr.  Libby  Rittenberg  (Department  of  Economics,  Colorado  College),  Dr. 

Robert  Durham  (Department  of  Psychology,  University  of  Colorado,  Colorado  Springs),  Dr. 

Donald  Bird,  (Department  of  Chemistry),  Mr.  David  Fitzkee,  (Department  of  Law) 

Abstract:  The  decade  of  the  90s  has  brought  with  it  a  dramatic  increase  in  emphasis  on  assessment  of 
student  academic  achievement  at  both  the  K-12  arid  higher  education  levels.  All  colleges,  universities, 
and  specialized  institutions  seeking  NCA  accreditation  have  been  required  to  invest  heavily  in  the 
development  and  implementation  of  assessment  programs.  During  this  session,  a  panel  of  educators 
and  assessment  specialists  from  the  Pikes  Peak  region  discussed  their  work  on  assessment  as  well  as 
their  opinions  about  progress  being  made. 

“...assessing  student  academic  achievement  is  an  essential  component  of  evaluating 
overall  institutional  effectiveness.  While  the  Commission  iraderstands  that  the  measurement 
of  learning  outcomes  is  only  one  aspect  of  a  total,  effective  educational  program,  it  recognizes 
that  assessment  data  contribute  to  successful  decision-making  within  an  institution, 
particularly  in  curriculum  and  faculty  development. 

“The  Commission’s  expectation  that  institutions  have  effective  programs  to  assess 
student  achievement  is  now  embedded  in  its  Criteria  for  Accreditation:  “The  institution  is 
accomplishing  its  educational  and  other  purposes”  (Criterion  Three)  and  “The  institution  can 
continue  to  accomplish  its  purposes  and  strengthen  its  educational  effectiveness”  (Criterion 
Four).  The  patterns  of  evidence  for  these  criteria  illustrate  the  centrality  of  an  effective 
assessment  program  in  documenting  the  effectiveness  of  an  institution’s  educational  programs 
and  the  institution’s  commitment  and  capacity  to  strengthen  those  programs.” 

North  Central  Association  (NCA) 

Handbook  of  Accreditation,  1997 

The  decade  of  the  90s  brought  with  it  a  dramatic  increase  in  emphasis  on  assessment 
of  student  achievement  at  both  the  K-12  and  higher-education  levels.  The  quote  above  from 
the  NCA’s  Handbook  of  Accreditation  illustrates  the  seriousness  of  this  change;  institutional 
accreditation  is  now  contingent  upon  successful  implementation  of  an  assessment  program. 
The  Air  Force  Academy,  like  nearly  all  other  higher-education  institutions,  has  invested 
considerable  time,  energy,  and  money  in  assessment.  Following  is  a  brief  summary  of  major 
initiatives  at  the  Academy  since  1992  that  illustrates  the  magnitude  of  our  investment. 

We  began  by  seeking  the  advice  of  a  number  of  nationally  eminent  experts  in  the 
academic  assessment  field  including  Dr.  Richard  Light,  Director  of  the  Harvard  Assessment 
Seminars;  Dr.  Jim  Nichols,  a  nationally-known  figure  in  implementing  assessment  programs; 
various  faculty  from  Alvemo  College,  umversally  considered  an  assessment  "showcase";  and 
Dr.  Peter  Ewell,  a  seminal  authority  on  assessment  in  the  American  Association  of  Higher 
Education.  Also  in  1992,  the  Superintendent  assembled  a  team  of  Academy  representatives 
and  senior  Air  Force  officers  to  create  new  vision  and  mission  statements  for  the  Academy.  It 
was  during  this  period  that  consensus  was  reached  on  the  Academy’s  core  values:  integrity 
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first,  service  before  self,  and  excellence  in  all  we  do.  Within  the  faculty,  the  Dean 
commissioned  a  group  of  senior  faculty  members  to  draft  desired  outcomes  of  the  academic 
program  at  USAFA.  In  1995,  in  response  to  a  requirement  from  NCA,  we  developed  a  plan 
for  assessing  the  academic  achievement  of  cadets  at  the  Academy.  The  plan  was  structured 
around  the  educational  outcomes. 

Our  assessment  work  was  not  limited  to  the  domain  of  academics.  Early  in  the  1990s, 
the  Air  Force  Chief  of  Staff  initiated  a  program  known  as  Quality  Air  Force.  Derived  from 
Total  Quality  Management  principles,  this  program  led  to  an  assessment  and  reporting 
process  called  Unit  Self  Assessment  (USA).  By  the  middle  of  the  decade,  each  major  Air 
Force  organization  was  creating  USA  reports.  The  Air  Force  Academy  completed  this  task  in 
1995.  The  format  for  the  report  followed  the  Malcolm  E.  Baldrige  criteria  and  so  the  process 
of  information  gathering  and  reporting,  in  many  ways,  was  similar  to  the  NCA  self  study. 

After  creating  the  NCA  Assessment  Plan  and  completing  the  institutional  USA, 
several  teams  were  formed  within  the  faculty  that  constituted  an  Educational  Outcomes 
Assessment  Working  Group  (EOAWG).  This  group  explored  an  assessment  philosophy  and 
techniques  that  would  be  well-suited  for  our  educational  outcomes.  Finally,  during  the  last 
several  years,  the  Dean  decentralized  assessment  responsibility  to  the  department  level  and 
each  department  has  written  a  unit  self  assessment  report.  Most  recently,  a  group  within  the 
faculty  has  created  a  catalog  of  assessment  techniques  that  are  currently  in  use  at  the 
Academy.  Every  academic  department  is  represented  and  the  catalog  contains  more  than  370 
entries.  The  Directorate  of  Education  (DFE)  continues  to  provide  assessment  support  through 
focus  groups.  Small  Group  Instructional  Diagnostics  (SGIDs),  and  expert  consulting  services 
when  requested. 

We  have  learned  a  number  of  lessons  from  this  work.  Some  of  the  more  important  ones 
are  : 

1 .  It  is  not  likely  that  there  is  a  “best”  way  to  assess  an  educational  outcome.  When  dealing 
with  complex  psychoeducational  issues,  we  must  rely  on  multiple  techniques  and  look  for 
patterns  of  convergence  in  the  data. 

2.  There  is  often  an  inverse  relationship  between  the  quality  of  assessment  methods  and  their 
expediency;  preferred  methods  usually  cost  more  and  take  longer  to  complete. 

3.  It  is  difficult  to  transfer  assessment  techniques  from  one  context  to  another;  the  only  way 
to  be  certain  it  works  in  your  program  with  your  students  is  trial  and  error. 

In  summary,  we  emphasize  three  points:  1)  assessment  programs  have  received  a  dramatic 
increase  in  attention  during  the  past  decade,  2)  most  higher-education  institutions  have 
invested  heavily  to  develop  successftil  programs,  and  3)  academic  assessment  has  proven  to 
be  complex  and  “expensive”  work.  Is  this  investment  making  a  difference? 

To  address  this  question,  we  assembled  a  panel  of  four  faculty  members  with  assessment 
expertise.  They  were  Dr.  Robert  Durham,  a  psychology  professor  at  CU-the  Springs,  Dr. 
Libby  Rittenberg,  an  economics  professor  at  Colorado  College,  Dr.  Don  Bird,  a  chemistry 
professor  at  the  Academy,  and  Nfr.  Dave  Fitzke,  a  professor  of  Law,  also  from  the  Academy. 
All  four  of  these  people  have  been  responsible  for  institutional  and/or  department-level 
assessment  activities  at  their  respective  institutions.  The  following  questions  were  used  to 
structure  the  panel  discussion: 


"  These  lessons  are  adapted  from  “A  Critical  Review  of  Student  Assessment  Options”,  by  Joseph  Prus  and  Reid 
Johnson,  Winthrop  College,  1992. 
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>  How  do  you  define  assessment? 

•  The  language  of  assessment  is  complex  and  seems  to  be  subject  to  differing 
interpretations  depending  on  the  disciplinary  background  of  the  practitioner.  How 
might  we  converge  on  a  common  understanding  of  assessment  concepts? 

•  Is  there  a  difference  between  your  concept  of  “assessment”  and  “measurement”? 

>  Can  you  give  an  example  (or  two)  of  effective  assessment  practice? 

•  How  would  you  define  “effective”? 

•  How  much  did  it  cost  (time,  manpower,  $) 

•  Can  you  cite  examples  of  decisions  that  were  significantly  informed  by  assessment 
data? 

•  How  can  we  transfer  best  practices  from  one  context  to  another? 

>  Can  you  give  an  example  of  a  poor  assessment  initiative  you  have  seen  or  experienced? 

•  Why  was  it  ineffective? 

•  How  can  we  learn  from  our  assessment  “failures”? 

>  Is  the  increased  emphasis  on  assessment  of  student  academic  achievement  making  a 
difference  in  higher-education? 

•  Are  learning  and  teaching  better  at  your  institution  than  a  decade  ago?  How  do  you 
know? 

•  To  what  degree  has  any  change  been  the  result  of  assessment  initiatives? 

It  is  not  possible  to  capture  the  rich  content  of  the  discussion  in  this  short  synopsis. 
However,  the  discussion  did  make  it  clear  that  our  understanding  of  assessment  is  still 
developing  and  that  all  three  of  the  institutions  represented  on  the  panel  have  examples  of 
success  as  well  as  some  growing  pains  in  their  assessment  work.  Is  assessment  of  student 
academic  achievement  making  a  difference  in  higher  education?  I  think  the  consensus  of  the 
panel  was  clearly  “yes”.  However,  it  is  work  filled  with  all  the  complexities  of  social 
science:  many  variables,  with  unclear  causal  relationships  among  those  variables;  dynamic, 
organic,  multi-level  systems;  and  a  high  degree  of  uncertainty  in  measurement.  As  we 
continue  to  refine  our  assessment  practices,  we  should  remember  the  words  of  Aristotle  who 
said:  “Our  treatment  will  be  adequate  if  we  make  it  as  precise  as  the  subject  matter  allows. 
The  same  degree  of  accuracy  should  not  be  demanded  in  all  inquiries. . .”  In  spite  of  these 
challenges,  assessment  of  student  academic  achievement  is  a  worthy  and  necessary  endeavor. 
Einstein  said,  “Not  everything  that  counts  can  be  counted,  and  not  everything  that  can  be 
counted  counts”.  It  may  never  be  possible  to  “count”  some  important  dimensions  of  student 
academic  achievement.  However,  if  our  inquiry  results  in  an  active  dialogue  about  important 
issues  and  an  improved  understanding  of  those  issues,  then  we  will  have  succeeded. 
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Abstract:  The  paper  reports  the  development  and  assessment  of  three  techniques  used  to  enhance 
learning  in  the  core  engineering  mechanics  course  here  at  the  USAF  Academy.  The  three  techniques 
are:  (1)  real  world  industrial  based  applications  designed  to  increase  student  motivation,  (2)  interactive 
multimedia  computer  visualizations,  and  (3)  hands-on  demonstrations  and  experiments.  Through  the 
use  of  three  different  types  of  assessment  techniques,  coordinated  with  students’  Myers-Briggs  and 
VARK  learning  styles  types,  we  have  arrived  at  a  variety  of  conclusions  which  are  described. 

Introduction 

Engineering  mechanics  education  is  undergoing  a  transformation  from  strictly  lecture- 
based  education  to  a  format  where  a  variety  of  innovative  learning  techniques  are  used.  This 
transformation  has  been  spurred  on  by  a  renewed  understanding  that  the  abstract  concepts 
inherent  in  engineering  mechanics  may  not  be  learned  effectively  in  a  predominantly  lecture 
based  environment.  Both  new  tools  for  enhancing  student  learning  as  well  as  concrete  data 
establishing  the  effectiveness  of  these  tools  are  needed.  This  paper  builds  on  previous  work 
using  innovative  teaching  tools  [1-8,1-15,19,20,24]  by  developing  and  assessing  our  current 
use  of  two  tools:  computer  based  visualizations  and  hands  on  demonstrations  and  . 
experiments.  These  tools  are  being  used  in  our  Fall  1998  Engineering  Mechanics  core  course 
which  is  taken  by  all  cadets  regardless  of  their  major. 

Evaluation  of  the  enhancement  in  student  learning,  brought  about  by  use  of  the  hands- 
on  and  visualization  tools,  has  been  accomplished  by  a  variety  of  assessment  techniques 
including  scores  on  quick  quizzes  taken  before  and  after  the  enhancement  tool  and  daily 
detailed  student  feedback.  The  daily  student  feedback  was  done  using  a  survey  which  took 
the  students  approximately  a  minute  to  complete  after  each  lecture.  The  survey  was  designed 
to  differentiate  among  four  things:  1)  student’s  interest  in  that  lecture’s  subject  matter;  2)  that 
day’s  learning  experience;  3)  student’s  ability  to  apply  material  covered  that  day;  and  4) 
student’s  interest  in  exploring  that  lecture’s  material  further.  The  quick  quizzes  are  designed 
to  measure  conceptual  understanding  of  certain  abstract  stress  oriented  concepts.  The  hands- 
on  tools  are  low-cost,  interactive  experiments  designed  to  enhance  understanding  of  specific 
abstract  concepts.  The  visualization  content  consists  of  finite  element  based  stress  results 
displayed  in  color  formats.  Both  the  hands-on  and  the  visualization  tools  are  designed  to 
emphasize  aspects  of  stress  analysis  which  our  students  have  traditionally  formd  difficult  to 
grasp.  The  results  from  the  surveys  and  quick  quizzes  are  correlated  with  the  student’s 
Myers-Briggs  Type  Indicator  (MBTI). 
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Hands-on  Content 

The  hands-on  content  for  the  study  involved  use  of  the  “student  opticon”  and  the 
“micro  torsion  tester”  shown  below  (Figures  1  and  2).  The  student  opticon  device  is 
composed  of  two  polarizing  filters  and  a  birefringent  photoelastic  material  beam  or  load  cell 
mounted  in  a  wooden  box.  Students  push  on  the  beam  or  load  cell  with  their  fingers  to 
produce  color  patterns  corresponding  to  changes  in  the  magnitude  of  maximum  shear  stress  in 
the  specimen.  The  device  also  enables  visualization  of  stress  distribution.  Materials  for  the 
device  cost  about  $30. 

The  hand-held  micro  torsion  tester,  in  combination  with  two  small  polarizing  filters, 
allows  the  student  to  apply  torque  to  a  circular  cross  section  and  see  the  resulting  stress 
distribution.  It  is  constructed  of  a  circular  wafer  of  birefringent  photoelastic  material  centrally 
sandwiched  with  clear  adhesive  between  two  pieces  of  clear  1/8  inch  thick  Plexiglas.  The 
Plexiglas  pieces  are  cut  with  a  circular  central  section  and  two  lever  arms  extending  in 
opposite  directions.  When  the  components  are  bonded,  the  upper  and  lower  Plexiglas  lever 
arms  are  offset  from  each  other  about  20°  so  that  the  student  may  squeeze  on  the  two  sets  of 
opposing  arms  to  produce  pure  torque  in  the  photoelastic  wafer.  When  students  applied  torque 
to  the  device,  the  color  patterns  indicating  changes  in  maximum  shear  stress  were  slightly 
distorted  due  to  bonding  imperfections.  However,  the  concentric  distribution  of  shear  stress 
over  the  cross  section  was  still  apparent. 


Figure  1.  Student  Opticon. 
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Figure  2.  Micro  Torsion  Tester. 


Visualization  Content 

Visualization  content  for  each  module  involved  several  slides  showing  finite  element 
based  color  stress  plots  illustrating  various  concepts  from  stress  analysis.  Real  world 
examples  were  used  as  the  context  for  the  visualization.  These  examples  entailed  brief 


Pre-operative  Loading 


Post-operative  Loading 


Figure  3.  Pre-  and  post-operative  loading  of  a  knee  joint. 


overviews  of  how  torsion,  bending,  and  combined  loading  applied  to  the  cases  of  turbine 
shafts,  aircraft  wings,  and  human  knee  joints  respectively.  For  example.  Figure  3  was  one  of 
the  slides  used  to  illustrate  a  knee  operation  which  was  proposed  to  change  combined  axial 
and  bending  loading  of  a  knee  joint  to  pure  axial  loading  aligned  with  the  mechanical  axis  of 
the  bone  [16].  This  real  world  example  was  followed  by  a  series  of  finite  element  based  stress 
plots  showing  various  concepts  intrinsic  to  combined  loading. 

Another  example  utilized  the  illustration  shown  in  Figure  4  where  the  distribution  of 
bending  stress  through  an  F-16  wing  cross-section  was  roughly  approximated  with  a  beam 
model. 
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Figure  4.  Visualization  of  normal  stresses  due  to  bending. 


neutral  plane. 
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The  One  Minute  Survey  (OMS) 

The  one  minute  survey  (OMS)  being  used  in  the  current  course  has  been  iteratively 
developed  over  the  last  three  semesters.  The  original  OMS  used  for  a  previous  study 
[Jensen2]  asked  only  for  MBTI  type  and  overall  lecture  rating.  In  order  to  gain  additional 
insight  into  the  effectiveness  of  the  modules,  a  refined  OMS  was  developed  and  used  for  the 
present  study.  The  refined  version  requests  information  about  the  students’  perception  of 
interest,  learning,  applicability  and  motivation  for  future  exploration  (see  Fig  5).  In  addition, 
the  MBTI  type  is  recorded.  This  OMS  was  given  after  each  lecture  and  took  about  a  minute 
for  students  to  complete.  Figure  5  shows  the  content  and  form. 


1  MINUTE  SURVEY  EM120  -  FALL  1998 
Lesson  #: _ 

MBTI  Type: _  VARK  Type: _ 

Please  rate  the  following  statements  on  a  scale  from 
1  to  10  (1  -  very  untrue;  10  -  very  true): 

_ 1.  Today’s  class  kept  me  interested. 

_ 2.  Today’s  class  was  a  good  learning  experience. 

_ 3.  This  class  prepared  me  well  to  apply  today’s 

concepts  to  problems. 

_ 4.  This  class  motivated  me  to  further  explore  today’s  concepts. 


Figure  5.  One  minute  survey  form. 


Table  1.  Number  of  standard  deviations  off  mean  (percentile)  for 
question  number  and  type. 


1  min.  Survey  Question 

S-TYPE 

N-TYPE 

Ql: 

-0.29 

0.12 

Lecture  was  interesting? 

(39) 

(55) 

Q2: 

-0.22 

0.49 

Lecture  helped  me  learn? 

(41) 

(69) 

Q3: 

0.00 

0.16 

Lecture  helped  me  to  apply  material? 

(50) 

(56) 

Q4: 

-0.88 

-0.22 

Lecture  motivated  me  to  explore  subject  further? 

(19) 

(41) 

As  can  be  seen  in  Table  1,  N-types  rate  the  module-based  lectures  higher  than  do  the 
S-types  for  each  of  the  four  questions  from  the  OMS.  This  is  not  an  expected  result  [8-10, 13, 
17,1 8, 21-23, 25].  In  previous  studies  with  hands-on  content  [Jensen2,4],  the  S-types  have 
responded  more  favorably  than  N-types  to  the  modules.  In  this  case,  however,  we  believe  that 
the  response  from  the  S  and  N-types  is  reversed  because  the  module-based  content  required 
the  students  to  abstractly  apply  the  content  contained  in  the  modules.  This  is  due  to  the  fact 
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that  the  modules  were  used  to  help  introduce  new  material,  which  the  students  knew  they 
would  need  to  be  able  to  use  to  solve  problems.  The  abstract  process  of  using  sensory 
information  to  formulate  problem  solving  strategies,  is  a  process  which  the  N-types  would 
view  more  favorably  than  would  the  S-types. 

Results  from  Quick  Quizzes 

Immediately  before  and  after  the  enhanced  learning  modules  were  presented,  a 
conceptual  quick  quiz  was  administered  to  measure  short-term  increase  in  understanding  as  a 
result  of  the  module.  The  results  of  the  quick  quizzes  were  compiled  for  S-type  and  N-type 
preferences,  and  for  all  students  without  regard  for  differences.  Table  2  shows  the 
improvement  in  the  percentage  of  correct  answers  on  the  quiz  questions  as  a  result  of  the 
material  presented  in  class. 


Table  2.  Measured  quick  quiz  improvement. 


Quick  Quiz 

Question 

S 

N 

All 

Torsion 

#1 

20.0% 

7.7% 

14.3% 

#2 

20.9% 

3.5% 

13.4% 

Bending 

#1 

20.0% 

15.0% 

23.1% 

#2 

32.0% 

30.0% 

32.7% 

Combined  Loading 

#1 

24.1% 

25.0% 

32.8% 

#2 

41.4% 

25.0% 

32.8% 

Average 

26.4% 

17.7% 

24.9% 

Although  a  rigorous  statistical  analysis  would  have  to  be  conducted  to  ensure 
statistical  significance,  it  does  seem  apparent  that  the  enhanced  learning  modules  do  provide 
additional  benefit  to  the  S-type  students  compared  to  the  N-types  and  the  general  population 
when  looking  at  the  average  results.  One  might  expect  the  hands-on  content  to  help  these 
students  the  most.  For  the  S-type  students,  the  greatest  benefit,  when  compared  with  the  “all” 
category,  was  found  in  the  torsion  and  combined  loading  modules.  The  N-type  students 
almost  always  showed  lower  rates  of  improvement. 

Conclusions 

The  present  work  has  focused  on  developing  and  assessing  two  learning  enhancement 
tools:  computer  based  visualizations  and  hands  on  demonstrations  and  experiments.  These 
tools  were  used  in  our  Fall  1998  Engineering  Mechanics  core  course  which  is  taken  by  all 
cadets  regardless  of  their  major.  Assessment  has  been  accomplished  by  use  of  two 
techniques:  1)  scores  on  quick  quizzes  taken  before  and  after  the  enhancement  tool  and  2) 
daily  detailed  student  feedback.  The  daily  student  feedback  used  a  survey  which  took  the 
students  approximately  a  minute  to  complete  after  each  lecture.  The  survey  asked  for 
feedback  in  four  specific  areas  for  each  lecture:  1)  student’s  interest  in  that  lecture’s  subject 
matter;  2)  that  day’s  learning  experience;  3)  student’s  ability  to  apply  material  covered  ^at 
day;  and  4)  student’s  interest  in  exploring  that  lecture’s  material  fhrtiber.  The  quick  quizzes 
were  designed  to  measure  conceptual  understanding  of  certain  abstract  stress  oriented 
concepts. 
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The  hands-on  devices  are  low-cost,  interactive  experiments  designed  to  enhance 
understanding  of  specific  abstract  concepts.  The  visualization  content  consists  of  finite 
element-based  stress  results  displayed  in  color  formats. 

The  results  fiom  the  surveys  and  quick  quizzes  were  correlated  with  the  student’s 
Myers  Briggs  Type  Indicator  (MBTI).  A  variety  of  results  were  obtained  using  the  two 
assessment  instruments.  From  the  surveys,  students  indicated  that  the  hands-on  and  visual 
content  overall  was  interesting,  enhanced  the  learning  experience  and  helped  them  to  solve 
problems.  However,  the  majority  of  students  did  not  find  that  the  module-based  lectures 
motivated  them  to  explore  the  day’s  content  further.  Surveys  further  indicated  that  the 
module-based  content  was  rated  highly  by  the  MBTI  “N”  type  students,  but  not  so  highly  by 
the  MBTI  “S”  types.  Although  this  runs  contrary  to  what  was  expected,  our  conjecture  is  that 
the  N-types  prefer  this  content  (and  S-types  do  not)  because  the  maimer  in  which  it  was 
presented  necessitated  that  the  conceptual  content  be  abstractly  applied  to  solve  problems. 

The  quick  quiz  gives  indication  of  the  short  term  gain  in  problem  solving  skills. 
Results  fiom  the  quick  quizzes  indicate  that  the  S-types  achieve  more  benefit  than  the  average 
student  or  the  N-types  fiom  the  module-based  content.  However,  larger  data  sets  would  be 
needed  in  order  to  ensure  accuracy  of  this  particular  result. 

Overall,  we  believe  that  the  project  provided  a  solid  foundation  in  terms  of 
development  of  content  and  assessment  strategies.  Significantly  more  work  needs  to  be  done 
in  order  to  obtain  modules  and  assessment  results  which  have  been  definitively  shown  to 
enhance  students  learning.  Continuation  of  this  work  is  planned.  Others  are  welcome  to  use 
our  modules  or  assessment  results  in  any  way  they  feel  is  appropriate.  To  obtain  these 
resources,  simply  contact  one  of  the  authors. 

This  work  was  sponsored  in  part  by  the  Air  Force  Office  of  Scientific  Research  and  by  NSF  under 
contract  DUE-97 51315. 
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Abstract:  Because  cognitive  learning  styles  affect  how  students  approach  new  material,  a  deeper 
comprehension  of  these  styles  can  help  faculty  enhance  student  learning  experiences.  In  this  paper  we 
discuss  how  learning  style  data  can  be  used  to  help  guide  student  study  habits  and  instructional 
strategies.  Using  the  Group  Embedded  Figures  Test,  Felder's  Index  of  Learning  Styles,  Kolb's 
Learning  Styles  Inventory  II  '85,  and  the  Kerisey  Temperament  Sorter,  we  examine  whether  or  not 
there  is  a  correlation  between  learning  style  and  academic  performance  in  an  introductory  computer 
science  course. 

Introduction 

A  student’s  learning  style  indicates  how  that  student  responds  to  a  wide  range  of 
intellectual  and  perceptual  stimuli  and  how  they  prefer  to  approach  new  material.  For 
example,  some  students  may  prefer  to  discuss  new  concepts  in  small  groups,  while  others  may 
prefer  solitary  study  of  those  concepts.  A  student’s  learning  style  can  therefore  be  used  to 
guide  the  student  to  the  study  techniques  that  are  most  likely  to  be  effective  for  them. 

We  can  also  use  learning  style  data  to  help  improve  instructor  presentations.  For 
example,  instructors  may  have  an  instructional  strategy  they  use  consistently  that  interferes 
with  a  particular  learning  style,  causing  students  with  that  learning  style  to  perform  less 
proficiently  than  other  students.  We  can  then  use  this  information  to  suggest  alternative 
delivery  methodologies  to  the  instructor  to  help  them  reach  those  students. 

Additionally,  research  has  shown  that  Aere  is  very  little  relationship  between  overall 
college  achievement  and  learning  style  [6],  yet  there  are  some  indications  that  there  is  a 
relationship  between  learning  style  and  performance  in  specific  subject  areas.  For  example, 
Moldafsky  found  that  learning  style  could  affect  an  individual’s  skill  in  information 
processing,  decision-making  attitudes  toward  computers,  and  computer  anxiety  [4].  We 
therefore  hypothesize  that  learning  style  may  affect  student  performance  in  particular  courses. 

All  students  attending  the  U.S.  Air  Force  Academy  ^SAFA)  are  required  to  take  an 
introductory  course  in  computer  science  (CompSci  1 10).  Because  the  course  is  taken  by  all 
students  in  either  their  fieshman  or  sophomore  year,  it  assumes  no  prior  knowledge  about 
computers.  The  course  has  been  designed  to  cover  four  basic  areas  of  study,  but  the  key  topic 
is  problem  solving  with  computers.  Since  students  need  to  know  how  to  solve  problems 
before  they  can  solve  them  using  computers,  we  start  by  helping  the  students  develop  their 
problem  solving  skills.  They  then  learn  how  to  use  these  skills  to  solve  problems  using 
computers  and  the  Ada  programming  language. 

Last  year  we  conducted  an  experiment  on  877  students  enrolled  in  an  introductory 
course  in  computer  science,  using  4  different  learning  styles  instruments  to  collect  learning 
style  data  on  those  students.  We  then  used  this  learning  style  data  to  recommend  suitable 
study  habits  for  those  students.  We  also  evaluated  (for  each  instructor)  the  impact  of  the 
various  learning  styles  on  8  measures  of  course  performance.  For  half  of  the  instructors,  we 
found  a  sufficiently  large  number  of  statistically  significant  results  to  indicate  that  discussion 
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Visual  and  Verbal  Learners 

Visual  learners  retain  more  from  things  they  see  -  pictures,  diagrams,  flow  charts,  etc.  Verbal  learners  get  more 
out  of  words  -  written  and  spoken  explanations.  If  you’re  a  visual  learner,  it  might  help  to  diagram  your  problem 
solutions  to  check  them  before  coding.  It  could  also  help  if  you  draw  a  picture  of  each  Ada  construct  you  learn.  If 
you’re  a  verbal  learner,  writing  the  required  English  algorithm  for  each  problem  solution  may  be  sufficient  for  you 
to  check  your  work. 

Figure  1.  Learning  Styles  Reading  Excerpt 


of  instructional  strategies  is  merited;  we  are  currently  holding  those  discussions  with  the 
instructors.  Finally,  we  conducted  course- wide  statistical  analysis  to  determine  the  impact  of 
learning  style  on  performance  in  this  course,  and  found  numerous  statistically  significant 
results. 

Guiding  Study  Habits 

One  of  the  uses  for  our  learning  styles  data  involves  guiding  student  study  habits 
based  on  that  data.  To  help  provide  this  guidance,  we  supplied  each  student  with  a  brief  (3 
pages)  reading  that  discusses  each  of  the  learning  style  models  and  relates  different  learning 
styles  to  the  approach  they  should  take  toward  the  material  in  CompSci  110.  Students  can 
then  use  this  reading  in  conjunction  with  their  learning  styles  instrument  results  to  select  their 
study  habits  appropriately. 

For  example,  one  of  the  dimensions  measured  by  Felder’s  ILS  is  the  visual/verbal 
dimension.  In  Figure  1,  we  provide  the  corresponding  excerpt  from  the  reading  provided  to 
the  students.  If  students  find  that  they  tend  toward  visual  learning,  they  can  use  this 
information  to  help  guide  their  study  of  Ada  constructs.  We  teach  them  how  to  use  control 
flow  graphs  (a  flowchart-like  representation)  to  graphically  depict  various  Ada  constructs, 
such  as  selection  and  iteration  constructs,  and  a  visual  learner  might  therefore  find  it  easier  to 
use  these  graphs  as  they  try  to  understand  the  constructs.  Alternatively,  verbal  learners  may 
find  that  using  this  graphical  representation  is  not  particularly  helpful.  These  students  may 
choose  to  use  syntax  boxes  for  the  constructs  or  textual  examples  to  help  them  rmderstand 
those  constructs. 

Instructor-Specific  Results 

We  can  also  use  learning  style  data  to  help  instructors  improve  their  instructional 
techniques.  For  instance,  if  an  instructor  uses  an  instructional  strategy  that  interferes  with  a 
particular  learning  style,  students  with  that  learning  style  may  exhibit  poor  performance  in  the 
course.  If  analysis  of  the  learning  style  data  indicates  this  to  be  true,  the  instructor  can  modify 
their  instructional  strategies  to  also  reach  those  students. 

To  see  how  we  can  use  this  information,  consider  one  instructor’s  results  for  Felder’s 
Active/Reflective  dimension.  The  correlations  for  this  instructor  are  positive,  relatively 
strong  (0.42  and  higher),  and  statistically  significant  for  5  of  the  response  variables,  indicating 
that  active  learners  tend  to  do  better  in  this  instructor’s  class  than  reflective  learners.  We  can 
then  use  this  information  to  suggest  that  the  instructor  build  more  “reflection”  time  into  his 
lectures,  trying  to  reach  a  better  balance  of  active  group  or  board  work  and  individual  work. 

Conclusions 

Because  learning  style  affects  how  a  student  responds  to  stimuli  and  approaches  new 
material,  there  are  a  number  of  ways  we  can  use  learning  style  data  to  enhance  that  student’s 
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learning  experience.  We  can  use  learning  style  data  to  guide  the  student  toward  more 
effective  study  habits  and  we  can  use  that  data  to  help  instructors  in  their  selection  of 
instructional  strategies. 

We  are  continuing  this  work  with  the  students  currently  enrolled  in  CompSci  110. 
Administering  the  learning  style  instruments  takes  a  reasonably  small  amoimt  of  time 
(typically  an  hour  or  so),  and  the  resulting  data  can  be  used  to  help  students  develop  their 
study  habits,  to  help  instructors  select  their  instructional  strategies  more  effectively,  and  to 
help  researchers  better  imderstand  how  different  learning  styles  can  affect  student 
performance. 

For  a  detailed  description  of  the  Learning  Style  Instruments  and  details  of  the 
statistical  analyses  accomplished,  please  see  the  researchers  for  a  more  extensive  written 
summary  of  the  work. 
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Abstract:  We  examined,  using  pre-  and  post-test  scores,  the  differences  in  student  performance  in 
economics  when  cadets  were  taught  using  two  different  teaching  methods.  In  addition,  we  examined 
the  theoretical  pitfalls  associated  with  using  pre-  and  post-test  data  without  careful  thought.  In 
addition  to  these  results,  we  examined  student  attitudinal  data  in  which  the  experimental  approach  to 
teaching  economics  is  preferred. 

Introduction 

In  pursuit  of  the  ever  illusive  perfect  pedagogical  technique  for  imparting  knowledge 
to  our  charges,  many  of  us  in  the  economics  profession  try  classroom  exercises  and 
demonstrations,  both  to  keep  our  students’  attention  and  to  demonstrate  that  there  may  be  a 
certain  element  of  validity  to  what  we  are  saying. 

Many  of  these  demonstrations  have  been  passed  by  word  of  mouth  from  generation  to 
generation  of  instructor  or  adapted  from  experiments  performed  in  a  laboratory  setting. 
Recently,  however,  some  consolidated  sources  for  using  experiments  in  economic  principles 
classes  have  appeared.  Most  notably,  the  classroom  experiment  book  by  Bergstrom  and 
Miller.  With  the  expanding  pool  of  resources  for  classroom  experiments  in  economics,  an 
examination  of  the  pedagogical  value  of  these  experiments  on  student  performance  is  in  order. 
Yet,  surprisingly  little  research  has  been  done  in  this  area.  It  has  been  claimed  for  years  that 
there  is  significant  added  value  in  interactive  learning.  Yet,  even  in  the  education  literature, 
there  has  been  surprisingly  paltry  examination  of  empirical  evidence,  particular  in  the  area  of 
comparing  the  performance  between  two  instructional  techniques  in  terms  of  added  cognitive 
value.  A  definitive  answer  caimot  convincingly  be  given  without  closer  examination. 

Recently  the  introductory  economics  curriculum  at  USAFA  made  a  transition  to  an 
experiment-based  approach  to  teaching.  In  doing  so,  we  conducted  pre-  and  post-testing  on 
the  students  who  were  randomly  selected  to  be  taught  using  either  the  experiment-based 
approach  or  a  traditional  lecture  methodology.  We  are  currently  in  the  process  of  examining 
the  differences  in  student  performance  between  the  two  teaching  techniques,  if  any.  In 
addition  to  the  performance  data,  we  have  data  on  student  demographics  and  previous 
academic  performance  measures.  Also,  we  have  data  on  student  attitudes  toward  the  different 
course  that  can  be  used  for  comparison  in  that  light. 

In  addition  to  examining  the  data  itself,  we  examine  issues  involved  in  assessment  of 
added  value  with  respect  to  test  score  comparison.  Many  of  the  issues  we  bring  up  have 
simply  been  ignored  in  previous  literature  on  assessment,  potentially  invalidating  much  of  the 
previous  research  in  this  area.  Simple  statistical  techniques  can  be  used  to  correct  the 
problems  present  in  previous  examination  of  this  kind  of  information. 

Preliminary  results  indicate  a  significant  added  value  to  student  knowledge  using  both 
teaching  techniques.  Each  methodology  seems  to  have  its  own  area  of  comparative 
advantage.  Results  such  as  this  can  easily  be  used  to  further  refine  economics  courses  in  the 
future  in  order  to  take  advantage  of  each  teaching  method’s  strengths. 
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This  workshop  was  intended  for  individuals  who  perform  or  supervise  behavioral,  physiological  or 
clinical  research  involving  human  subjects.  The  workshop  recognized  the  need  of  the  researcher  to 
navigate  through  the  human  subjects  review  process  as  quickly  and  efficiently  as  possible.  Emphasis 
was  placed  on  educating  researchers  on  the  federal  requirements  for  the  protection  of  human  subjects 
in  research,  describing  the  USAFA  Institutional  Review  Board  process,  describing  shortcuts  to  speed 
the  review  process,  describing  pitfalls  that  typically  delay  research  approval,  and  addressing  questions 
frequently  encountered  from  researchers.  The  goal  was  to  provide  workshop  attendees  with  practical 
information  on  how  to  complete  requirements  for  human  subjects  review  of  their  research. 

Institutional  Review  Board 

❖  The  IRB  is  charged  to  protect  the  rights  and  welfare  of  human  subjects  in  research 

>  Authority  and  requirements  for  USAFA  IRB  derive  from  Federal  law,  DoD  &  AF 
policy 

^  IRB  has  the  authority  to  approve,  require  modifications  or  disapprove  human  research 

>  Approved  research  is  subject  to  continuing  and  armual  IRB  review 
.  ❖  The  administrative  agent  for  the  USAFA  IRB  is  USAFA/XPR 

The  IRB  Process 

❖  Principal  investigator  (PI)  prepares  package  using  protocol  outline  and  informed  consent 
document  template 

❖  PI  submits  for  department  coordination 

❖  Preliminary  review  by  IRB  Administrator  (USAFA/XPR) 

❖  IRB  review 

❖  PI  completes  modifications,  if  required 

❖  Approval  by  USAFA/CV 

❖  Approval  by  HQ  AFMOA 

❖  Begin  data  collection! 

IRB  Review  Process 

❖  Protocols  distributed  to  IRB  before  meeting 

>  PI  may  be  contacted  by  reviewers  with  questions  and  requests  for  clarification 

❖  Protocol  summarized  at  IRB  meeting  followed  by  discussion  and  vote 

>  Protocol  may  be  approved,  approved  with  modification  or  disapproved 
♦♦♦  Board  meetings  are  open  and  Pis  are  encouraged,  but  not  required,  to  attend 
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Criteria  for  IRB  Research  Review 

❖  Risks  to  subjects  are  minimized  by  using  sound  research  design 

❖  Risks  to  subjects  are  reasonable  in  relation  to  anticipated  benefits  to  subjects  and  the 
importance  of  the  knowledge  to  be  gained 

❖  Selection  of  subjects  is  equitable 

❖  Informed  consent  is  sought  and  documented 

❖  Research  provides  adequate  provisions  for  safety  of  subjects  and  monitoring  of  data 

❖  Research  provides  adequate  provisions  to  protect  the  privacy  of  subjects  and  maintain 
confidentiality  of  data 

❖  Safeguards  are  in  place  to  protect  the  rights  and  welfare  of  “vulnerable”  research 
populations 

Informed  Consent 

❖  Assures  that  subjects  understand  the  nature  of  the  research  and  can  knowledgeably  and 
voluntarily  decide  to  participate 

❖  Cadets  present  a  “doubly”  vulnerable  research  population;  subject  to  undue  influence  as 

>  Student:  Student-teacher  relationship,  grades,  competitive  selection 

>  Military  member:  Cadet-officer  influences,  upperclass-underclass  influences,  class 
loyalty 

Criteria  for  Informed  Consent  Review 

❖  Description  of  study:  Purpose,  duration,  procedures,  compensation/costs 

❖  Description  of  risks:  Reasonably  foreseeable  risks,  discomforts 

❖  Description  of  benefits:  Direct  benefits  to  subject 

❖  Disclosure  of  alternatives:  Same  benefits/treatments  available  elsewhere? 

❖  Extent  of  confidentiality:  How  will  data  be  protected? 

❖  Compensation  or  treatment  for  injury:  Only  for  studies  with  greater  than  minimal  risk 

❖  Contact  for  questions:  Research,  subject  rights,  research  injury 

❖  Voluntary  participation  statement:  Can  withdraw  at  any  time  without  loss  of  benefits 

Tips  to  speed  IRB  approval 

❖  Identify  problem  areas  through  early  discussions  with  IRB  members  and  experienced 
researchers 

❖  Use  outlines  and  templates  from  human  subjects  web  page 

>  These  satisfy  Federal  requirements  and  are  updated  frequently 

❖  Use  minimum  number  of  subjects  needed  to  meet  research  goals 

>  Justify  subject  requirements  with  power  analysis  or  prior  literature 

Common  IRB  Pitfalls 

❖  Using  technical  jargon  in  the  protocol  and  the  informed  consent  document 

❖  Introducing  oral  changes  at  the  IRB  meeting 

❖  Failure  to  include  subject  recruitment  material 

❖  Using  only  subject  populations  of  “convenience” 

❖  Failure  to  recognize  sources  of  undue  influence  in  subject  recruitment,  selection  and 
participation 

❖  Inadequate  or  over-extended  data  protections 
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Keeping  data  for  longitudinal  research  without  consent 


After  Your  Protocol  is  Approved 

❖  The  PI  ensures  that  the  research  complies  with  human  subjects  protections  and  is 
conducted  lAW  approved  protocol 

>  Modifications  require  IRB  approval 

>  Monitor  research  and  activities  of  associate  investigators 

>  “Adverse  events”  reported  immediately  to  the  IRB 

>  Significant  changes  to  risk/benefit  reported  to  the  IRB 

❖  Provide  progress  and  final  reports 

❖  Submit  signed  informed  consent  documents  to  XPR  (for  permanent  archiving)  after 
completion  of  research 

IRB  Resources 

❖  Guidelines  and  instructions  available  from: 

>  USAFA  IRB  web  site:  http ://www.usafa. af mil/dfe/herc/subi ects .htm 

>  USAFA  IRB  Administrator  (XPR,  719-333-2587)  or  IRB  Chair  (Dr  Miller,  719-333- 
2804) 

>  IRB  members.  Department  research  directors  and  mentors,  and  Federal,  DoD  and  AF 
policy  documents  (policy  documents  at  USAFA  IRB  web  site) 
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Svirveys  -  Design,  Administration,  Analysis,  Utility,  Lessons  Learned,  Minimums,  Better  and 

Bests  (Workshop) 

Leray  L.  Leber,  Ph.D.,  Lt  Colonel,  USAF 
Bernard  Asiu,  Ph.D.,  Lt  Colonel,  USAF 
Office  of  Institutional  Research 

Abstract:  Optimizing  the  utility  of  survey  results  was  the  primary  focus  of  this  workshop.  The 
presenters  have  authored,  co-authored,  administered,  analyzed,  and  given  feedback  on  dozens  of 
surveys  over  the  past  year  to  groups  as  small  as  25  to  as  large  as  225,000.  Real-life  successes  and 
failures  reinforced  many  survey  issues  to  include  establishing  timelines,  cost-benefit  tradeoffs, 
protecting  respondent  privacy,  optimizing  participation,  sampling  strategies,  and 
analysis/communication  of  results. 

Survey  Planning 

❖  Goal:  Establish  survey  needs 

>  Determine  data  needs 

>  Establish  data  constituents 

>  Develop  timetable 

>  Evaluate  past  experience/instruments/data 

Survey  Development 

❖  Goal:  Translate  data  needs  into  survey  items 

>  Develop  items  and  validate  to  objectives 

>  Draft  the  survey  instrument 

>  Establish  administration  methods 

>  Coordinate  with  impacted  offices 

Survey  Piloting 

❖  Goal:  Validate  methodology  and  instrument 

>  Administer  the  pilot  instrument 

>  Evaluate  pilot  data 

>  Validate  items,  methodology  and  analysis  plan 

>  Implement  changes  and  lessons  learned 

Survey  Administration 

❖  Goal:  Facilitate  survey  experience  and  obtain  quality  data 

>  Logistic  preparation 

>  Improving  data  quality  through  administration  and  participation 

>  Data  protection 

>  Respondent  and  customer  follow-up 
Survey  Analysis 

❖  Goal:  Provide  data  that  meet  customer  needs 

>  Establish  customer  interest  and  sophistication 

>  Determine  kinds  of  data  to  be  provided 

>  Descriptive  vs.  inferential  statistics 
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>  Responses  from  open-ended  items 

>  Effect  sizes 

>  Ancillary  findings  from  data 

Reporting  Survey  Results 

❖  Goal:  Provide  survey  results  in  a  usable  format 

>  Establish  presentation  medium  (report,  talker,  presentation,  etc.) 

>  Organize  results  according  to  customer  priorities 

>  Communicating  complex  statistical  results 

>  Aiding  interpretation  of  results 

>  Providing  recommendations 

After  the  Survey 

❖  Goal:  “Close  the  loop” 

>  Provide  feedback  to  survey  participants 

>  Disseminate  survey  results 

>  Assess  survey  effort—  improvements  to  instrument,  administration,  analysis  and 
reporting 

>  Support  customer  “change  team” 

Top  10  Survey  Mistakes 

❖  Assure  yourself  that  the  same  mistakes  won’t  happen  again. 

❖  The  longer  the  survey,  the  better  it  Avill  be. 

❖  Make  the  survey  your  own,  and  only  your  own. 

❖  Make  sure  the  survey  is  unexpected. 

❖  Piloting  is  worthless,  the  survey  works  for  me. 

❖  Survey  administration:  What  me,  worry? 

❖  More  emalysis  is  better,  leave  no  stone  imtumed. 

❖  Results:  Better  right  than  when  needed. 

❖  Keep  your  results  quiet. 

❖  You’re  done  when  the  survey’s  picked  up  and  the  lights  are  turned  out. 

Survey  Resources 

❖  Babbie,  Earl  R.  (1990).  Survey  research  methods.  Belmont  CA:  Wadsworth  Inc. 

❖  Babbitt,  Bettina  A.,  &  Nystrom,  Charles  0.  (1989).  Questionnaire  construction  manual. 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA 
(DTIC  Accession  Ntimber  AD-A212-365  &  AD-A213-255). 

❖  Converse,  Jean  M.,  &  Presser,  Stanley  (1986).  Survey  questions:  Handcrafting  the 
standardized  questionnaire.  Newbury  Park,  CA:  SAGE  Publications  Inc. 

❖  DeVellis,  Robert  F.  (1991).  Scale  development:  Theory  and  applications.  Newbury  Park, 
CA:  SAGE  Publications  Inc. 

❖  Fowler,  Floyd  J.  (1988).  Survey  research  methods.  Newbury  Park,  CA:  SAGE 
Publications  Inc. 

❖  McNamara,  James  F.  (1994).  Surveys  and  experiments  in  education  research.  Lancaster 
PA:  Technomic  Publishing  Inc. 
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Rossi,  Peter  H.,  Wright,  James  D.,  &  Anderson,  Andy  B.  (Eds.)  (1983).  Handbook  of 
survey  research.  San  Diego:  Academic  Press. 


Writing  for  Publication  (Workshop) 


Barbara  J.  Millis,  Ph.D. 

Faculty  Development,  Directorate  of  Education 

Abstract:  In  this  workshop  faculty  members  learned  basic  writing,  editing,  and  targeting  strategies  to 
help  their  work  reach  a  wider  audience.  This  workshop  dealt  with  issues  such  as  getting  started, 
selecting  appropriate  journals,  soliciting  assistance,  and  preparing  the  manuscript,  including  editing 
techniques.  Participants  were  encouraged  to  share  "works  in  progress"  with  the  idea  that  this  initial 
meeting  would  be  only  the  beginning  of  supportive  help  from  USAFA  colleagues.  The  facilitator 
shared  a  model  for  a  "writers'  support  group"  active  at  Keane  College. 

"Writing  is  a  way  to  learn,  not  merely  a  means  of  communicating  to  others  what  has  already 
been  mastered.  It  is  a  critical  tool  of  invention  and  discovery  central  to  all  disciplines." 

Elaine  Maimon 

Although  the  United  States  Air  Force  Academy  prides  itself  on  its  teaching  mission, 
research  also  plays  a  critical  role  here.  Regardless  of  one's  discipline,  research  involves 
dissemination,  sharing  the  results  with  others  through  conference  presentations  and  scholarly 
papers.  Invariably,  this  dissemination  requires  putting  pen  to  paper  or  more  likely  now, 
fingers  to  keyboard.  Given  the  demands  on  our  time  at  USAFA,  producing  first-rate  journal 
articles,  books,  or  conference  presentations/proceedings,  can  be  a  daunting  task. 

Some  of  these  challenges  are  identified  in  Keimeth  Henson's  (1995)  interesting  list  of 
“Six  Myths  that  Haunt  Writers,”  which  focuses  specifically  on  academic  writing:  (1)  I'm  not 
sure  I  have  what  it  takes;  (2)  I  don't  have  time  to  write;  (3)  I  don't  have  anything  worth  writing 
about;  (4)  The  editors  will  reject  my  manuscript  because  my  name  isn't  familiar  to  them;  (5) 
My  vocabulary  and  writing  skills  are  too  limited;  and,  (6)  In  my  field  there  are  few 
opportunities  to  publish. 

The  first  step  for  some  of  us  is  learning  to  identify  and  overcome  some  of  these 
writing  inhibitors. 

Causes  of  Writing  Problems 

"What  can  I  do  in  Poetry/  Now  that  the  good  Spirit's  gone  fi'om  me?/Why  nothing 
now,  but  lonely  sit, /And  over-read  what  I  have  writ."  — ^Robert  Herrick 

As  this  jingle  by  a  seventeenth-century  poet  suggests,  writer's  block  is  nothing  new. 
We  can  imagine  the  earliest  cuneiform  scribes  laying  out  their  chisels,  much  as  Ernest 
Hemingway  laid  out  his  sharpened  pencils.  Robert  Boice  (1990)  has  identified  some  causes 
of  writing  problems  that  might  help  would-be  authors  identify  some  of  the  causes  of  potential 
paralysis.  Some  of  these,  such  as  fear  of  failure,  lack  of  confidence,  or  a  tendency  toward 
perfectionism,  are  internally  generated.  The  first  two  can  prevent  a  would-be  writer  from 
starting  a  project  at  all;  the  latter  can  result  in  an  inability  to  finish  writing  or  an  imwillingness 
to,  as  Tara  Gray  (1998)  puts  it,  "Kick  it  out  the  door — and  make  'em  say  'no.'"  Sometimes 
these  inhibitors  may  occur  because  of  early  negative  writing  experiences,  such  as  vivid 
imagines  of  Mrs.  Grundy  squinting  over  granny  glasses  at  her  cowering  pupils,  bun  pulled 
tightly  back  and  ruler  in  hand.  Or,  they  may  result  from  a  desire  to  avoid  the  irritability  or 
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obsessiveness  that  Boice  suggests  can  arise  from  "binge  writing,"  writing  frenetically  in 
huge,  unhealthy  chunks  of  time. 

Sheer  procrastination  probably  tops  everyone's  list  of  writing  problems.  Given  our 
busy  schedules  and  our  commitment  to  open  door  policies,  it  is  easy  to  postpone  that  first 
draft  in  favor  of  extra  instruction  or  a  required  committee  meeting.  Boice  mentions  The  Law 
of  Delay  ("That  which  can  be  delayed,  will  be")  with  the  assertion  that,  "The  priority  principle 
simply  says  that  if  you're  not  writing  as  much  as  you  would  like,  you're  not  making  writing  a 
realistic  priority"  (p.  75). 

Despite  the  obstacles  Boice  and  Henson  identify,  some  specific  approaches  seem  to 
help  faculty  members  become  productive  writers. 

Overcoming  Writer's  Block 

“The  Priority  Principle:  That  which  can  be  delayed,  need  not  be.”  — ^Robert  Boice 

To  get  "unstuck,"  virtually  all  writers  recommend  a  technique  called  "free  writing" 
advocated  by  Peter  Elbow  (1998;  1998).  Free  writing  is  simply  a  brainstorming  technique 
where  one  writes  rapidly  and  nonjudgmentally  about  ideas  related  to  the  topic.  Elbow 
recommends  writing  continuously  for  ten  minutes,  even  if  the  product  seems  pointless  or 
nonsensical,  such  as,  "This  is  a  stupid  activity.  I'll  never  get  this  article  finished."  Debbie 
Ridpath  Ohi  (1998)  puts  it  this  way:  "The  best  thing  to  do  is  to  brainstorm,  which  is  write 
down  whatever  comes  to  mind  about  the  subject  you  are  working  on.  It  may  not  help  you 
immediately,  but  it  gives  you  ideas,  which  lead  to  even  more  ideas.  Writing  stream-of- 
consciousness  style  will  usually  reveal  the  thoughts,  topics,  inspiration  that  your  brain  is 
withholding  from  you." 

Boice  suggests  uses  "free  writing"  as  the  foundation  of  an  approach  he  calls 
“generative  writing.”  After  the  brainstorming  period,  he  suggests  reflection,  an  editing  phase 
where  writers  combine  similar  ideas  and  refine  prose  for  consistency  in  direction,  voice,  and 
audience.  The  sequence  is  repeated  as  often  as  it  takes  to  complete  a  draft. 

Many  writers  employ  techniques  that  work  specifically  for  them,  such  as  taking  a  brief  walk 
or  listening  to  music.  For  scholarly  work,  two  approaches  work  well:  (1)  Some  writers  have 
learned  to  jump  ahead  to  write  a  section  of  the  book  or  article  that  they  know  best.  Focusing 
on  the  easy  parts  gets  the  creative  ideas  flowing;  and  (2)  Reviewing  research  notes  and 
rereading  what  is  already  written  can  help  ideas  to  "jell." 

All  successful  writers  encourage  others  to  write  often  and  regularly.  Boice  urges  us, 
for  example,  to  "Abandon  the  notion  that  writing  is  best  done  in  large,  undisrupted  blocks  of 
time.  Write  in  small  regular  amounts  when  you  are  most  fresh.  Work  on  'doable'  blocks, 
specific,  finishable  tmits  of  writing  each  session"  (p.  77).  This  is  probably  the  hardest  advice 
of  all.  For  academic  researchers,  it  is  also  essential  to  establish  a  scholarly  mind  set. 

Establishing  a  Scholarly  Mind  Set 

Learn  to  swallow  criticism  whole,  “the  way  an  owl  eats  a  mouse.”  --Peter  Elbow 

Research  writing  strategies  never  come  in  a  one-size-fits  all  format.  But  here  are  some 
proactive  things  that  any  of  us  can  do: 

•  Surround  ourselves  with  the  academic  resources  we  need.  We  must  be  certain  that 
the  library,  our  department,  or  our  household  receives  appropriate  books,  journals 
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or  whatever  else  we  need  to  stay  current  in  our  fields.  Personal  expenditures  may 
be  necessary:  fortunately,  they  are  tax  deductible. 

•  Foster  "cutting  edge"  contacts  or  relationships.  Conferences,  Listservs,  former 
colleagues,  and  our  Air  Force  affiliations  can  provide  opportunities  for  critical 
networking. 

•  Practice  the  art  of  scholarly  reflection.  We  should  always  be  on  the  look  out  for 
that  new  angle,  that  rmexpected  insight,  or  that  blinding  period  of  creative 
thinking.  We  must  try  to  RECORD  everything,  whether  in  a  journal,  on  index 
cards,  or  in  our  computer  databases. 

•  Strike  while  the  iron  (or  the  brain)  is  hot.  Following  this  advice  may  mean  for 
some  of  us  nocturnal  visits  to  the  computer,  a  pu22ling  practice  to  significant 
others,  offspring,  and  family  pets. 

•  Despite  listening  to  our  muse  when  it  hits  us  over  the  head,  we  can't  always  rely  on 
inspiration.  Developing  productive  work  habits  is  essential.  _  - 

•  Think  in  terms  of  multiple  publications,  spin-offs,  connections,  or  different 
journals.  Results  of  a  scholarly  study  published  in  a  research  journal  might  have 
practical  classroom  applications,  for  example,  that  could  be  shared  through  less 
formal  publications  such  as  College  Teaching. 

•  Self-congratulation  is  not  enough.  We  need  to  take  advantage  of  peer  review 
options  and  opportunities  by  enlisting  advice  from  our  colleagues  here  and  at  other 
institutions. 

This  last  idea  is  probably  the  most  critical.  Boice  calls  helping  others  "missionary 
work,"  a  term  that  resonates  well  with  our  core  value  of  "Service  before  Self"  A  second  pair 
of  eyes — and  preferably  more! — can  give  us  new  perspectives  and  help  us  put  ourselves  in  the 
place  of  the  reader.  He  recommends:  “Share  your  writing  with  supportive,  constructive 
firiends  before  you  feel  ready  to  go  public.  If  other  readers  know  they’re  being  asked  to 
appraise  vwriting  in  its  formative  stages,  they’ll  feel  less  judgmental  and  more  inclined  to  offer 
advice  for  changes”  (p.  78). 
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Fitting  Courses  Together  to  Make  a  Better  Curriculum 

John  L.  Putnam,  Ph.D.,  Major,  USAF 
Robert  K,  Noyd,  Ph.D. 

Department  of  Biology 

Abstract:  Our  paper  describes  a  process  the  USAFA  Biology  Department  is  using  to  review  and 
create  a  new  curriculum.  The  final  product  will  be  a  collection  of  "course  charters,"  which  describes 
the  department's  expectations  for  required  courses  in  the  major.  With  expectations  outlined,  each 
course  fits  together  to  form  an  integrated  curriculum. 

Rationale  for  Redesigning  Our  Curriculum 

Introduction.  We  have  been  operating  with  an  "implicit  curriculum."  As  a  loosely 
knit  collection  of  courses,  the  implicit  curriculum  simply  listed  required  courses,  stated 
prerequisites,  and  suggested  a  course  sequence.  As  a  result,  the  department's  "product"  (the 
student  major)  and  educational  goals  were  ill  defined,  vague,  and  were  not  readily  apparent  to 
administrators,  faculty,  or  students.  Without  a  clear  set  of  educational  goals,  it  was  difficult  to 
assess  our  curriculum's  strengths  and  weaknesses.  More  importantly,  there  was  no  sound 
basis  for  making  important  curriculum  decisions,  improvements,  or  changes. 

Problems  with  the  Implicit  Curriculum.  For  us,  the  implicit  curriculum  has  produced 
several  problems  and  issues  at  the  course  level.  The  implicit  curriculum  left  circumscription 
and  development  of  content  and  skills  entirely  to  instructors.  Thus,  each  course  evolved 
along  its  own  path  and  the  curriculum  was  simply  a  sum  of  all  courses  in  their  particular  stage 
of  development.  The  fact  remained  that  courses  did  not  operate  independently;  upper  level 
courses  depended  on  the  content  and  skills  of  introductory  courses.  Furthermore,  the  lack  of 
curriculmn  integration  and  cohesion  lead  instructors  to  assume  that  students  came  to  their 
classes  with  a  specific  set  of  skills  (i.e.  writing,  statistical)  or  a  depth  of  content  knowledge. 
These  assumptions  arose  fi-om  a  poor  understanding  of  the  content  areas,  the  skills  developed, 
or  the  projects  required  by  other  courses.  For  example,  instructors  of  our  senior  seminar 
course  expected  that  students  came  with  the  ability  to  write  in  a  scientific  paper  format  and 
could  analyze  a  paper's  strengths  and  weaknesses.  Upon  examination,  we  found  that  students 
had  never  been  required  to  do  this  prior  to  the  senior  seminar  course,  leaving  a  gap  in  their 
background.  We  also  found  content  areas  between  courses  that  essentially  repeated  the  same 
material.  Problems  associated  with  a  lack  of  course  integration  increased  with  retirements 
and  faculty  turnover  when  new  instructors  were  left  with  little  guidance  in  the  role  of  their 
courses  in  the  curriculum  and  specific  content  areas  to  address. 

Defining  Curriculum.  We  directly  confronted  these  issues  by  attempting  to  fit  courses 
together  to  create  an  integrated  curriculum,  one  that  specifies  its  goals  and  defines  its  product. 
We  viewed  the  product  of  our  curriculum  developmental  efforts  to  be  a  set  of  general 
knowledge,  skills,  and  attitudes  (KSAs)  we  intend  to  develop  in  our  students  on  a  department 
basis  and  specific  outcomes  on  a  course-by-  course  basis.  Ultimately,  each  required  course 
has  a  specific  role  in  the  curriculum,  a  set  of  KSAs  assigned  to  it  by  the  curriculum.  We  call 
this  document  of  expectations  a  course  charter.  Inherent  in  the  charter  concept  is  a 
philosophy  that  a  course  serves  the  curriculum. 
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^'-'-''Educational  Goals 

^sKnowledge  Skills  Attitudes^ 

Course  Charter  1 

Course  Charter  2 

Course  Charter  3 

•  K  outcomes 

•  K  outcomes 

•  K  outcomes 

•  S  outcomes 

•  S  outcomes 

•  S  outcomes 

•  A  outcomes 

•  A  outcomes 

•  A  outcomes 

Figure  1.  The  process. 


Building  a  New  Curriculum  --  the  Department  of  Biology's  Process 

Introduction.  Our  curriculum  has  a  top  down  structure  (Figure  1,).  At  the  top, 
outlining  the  curriculum,  are  the  department's  educational  goals.  The  educational  goals 
describe,  in  broadest  terms,  the  knowledge,  skills  and  attitudes  we  want  our  required  courses 
to  instill  into  our  biology  majors.  Below  the  goals  we  place  our  course  charters-the  heart  of 
the  curriculum.  Each  charter  is  a  listing  of  outcomes,  and  describes  the  educational  intent  of 
the  course.  Each  outcome  is  actually  a  sub-category  of  an  educational  goal,  and  thus  is  linked 
to  the  over  riding  guiding  principles.  Fimctionally,  the  outcomes  define  more  specifically  the 
knowledge,  skill,  or  attitude  the  course  is  charged  with  passing  on  to  our  students.  Our 
challenge,  in  building  a  curriculum,  was  to  define  our  educational  goals,  create  outcomes,  and 
assemble  them  into  course  charters. 

Defining  the  Educational  Goals^  One  faculty  member  researched  and  drafted  a  set  of 
educational  goals.  As  a  faculty,  we  then  clarified  and  validated  the  goals  in  a  series  of  small 
group  meetings  that  promoted  discussion  and  proved  to  be  quite  productive.  A  validation  of 
the  goals  was  critical  to  creating  ,  and  prevent  the  appearance  of  micro-management  from 
leadership.  Without  the  validation  of  the  faculty,  members  may  have  viewed  the  goals  as 
dictates  from  on  high. 

Creating  Outcomes.  With  goals  written,  the  next  step  involved  writing  outcomes  for 
each  goal  (a  knowledge  area,  skill,  or  attitude).  We  formed  seven  small  committees  of  3-4 
faculty  members.  All  faculty  members  participated.  We  asked  each  committee  to  write 
outcomes  for  a  single  goal.  To  ensure  timeliness,  we  assigned  a  facilitator  to  each  committee, 
established  a  reasonable  timeline  with  deadlines,  and  provided  written  guidance  on  writing 
outcomes.  To  increase  the  likelihood  of  a  faculty  wide  consensus  on  the  outcomes,  we 
encouraged  inter-committee  communication  by  publishing  each  committee’s  work.  Thus, 
members  of  each  committee  could  read  and  provide  feedback  on  all  the  outcomes.  In 
addition,  we  held  three  small-group  meetings  (one  member  from  each  of  the  seven 
committees)  to  discuss  the  outcomes.  Using  small  committees  to  write  our  outcomes  eased 
logistics,  and  permitted  us  to  tackle  many  goals  simultaneously.  Input  from  the  entire  faculty 
created  synergism,  and  increased  buy-in  to  the  project. 

Assembling  Charters.  After  writing  their  outcomes,  and  gaining  approval  from  the 
department,  the  committees  decided  which  specific  required  course  should  address  each  of 
their  outcomes.  Thus,  we  did  not  create  courses  to  carryout  our  outcomes.  Rather,  we  used 
the  required  courses  that  currently  constitute  our  curriculxim.  In  addition  to  assigning  the 
outcomes  to  courses,  the  committees  determined  the  degree  of  emphasis  it  expected  for  each 
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outcome.  It  indicated  the  degree  of  emphasis  by  assigning  each  outcome  a  designator  from  a 
three-tiered  scaled.  Thus,  course  directors  have  some  indication  of  the  relative  importance  of 
each  outcome  in  their  course  charter.  To  assemble  a  charter  for  a  single  course,  we  simply 
listed  all  the  outcomes  assigned  to  the  course.  To  validate  the  reasonableness  of  the  each 
committee's  charter,  we  are  now  asking  the  current  course  instructors  to  review  the  charter. 

Other  Considerations 

•  Communications.  We  found  it  vital  to  maintain  communications  with  the  faculty.  To  that 
end  we  created  a  newsletter.  We  used  the  newsletter  to  address  changes  in  processes, 
announce  meetings,  and  publish  the  work  of  the  committees. 

•  Common  Terminology.  For  the  faculty  to  cooperate  in  this  venture  they  needed  to  have  a 
common  vocabulary  of  curriculum  development.  We  published  a  glossary  of  terms,  and 
explained  terms  in  our  start-up  meeting.  Thus,  faculty  could  discuss  outcomes  and  goals 
without  confusion. 

•  Establishing  a  Time  Frame.  Long  term  projects  demand  concrete  end  point,  other  wise 
they  never  come  to  completion.  In  addition  to  defining  an  end,  we  established  milestones 
to  keep  the  project  moving  forward. 
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